[jira] [Commented] (KAFKA-4570) How to transfer extended fields in producing or consuming requests.

[dongeforever (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880002#comment-15880002
 ] 

dongeforever commented on KAFKA-4570:
-

[~ewendland] it seems that I can use sasl to authenticate the channel and ssl 
to encrypt the data.  

For every user, they may provider their user name and password, and the broker 
sasl server to verify it.

> How to transfer extended fields in producing or consuming requests.
> ---
>
> Key: KAFKA-4570
> URL: https://issues.apache.org/jira/browse/KAFKA-4570
> Project: Kafka
>  Issue Type: Wish
>  Components: clients, core, security
>Affects Versions: 0.10.1.1
>Reporter: dongeforever
>Priority: Critical
>  Labels: features
>
> We encounter a problem that  we can not transfer extended fields for 
> producing or consuming requests to the broker.
> We want to validate  the producers or consumers in a custom way other than 
> using SSL.
> In general, such as JMS, it is possible to transfer user-related fields to 
> server.
> But it seems that Kafka dose not support this, its protocol is very tight and 
>  unable to add user-defined fields.
> So is there any way  achieving this goal ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Manikumar]

+1 (non-binding)

On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat  wrote:

> Hi Jun,
>
> Thanks a lot for the comments and reviews.
> I agree we should log the username.
> What I meant by creating KafkaPrincipal was, after this KIP we would not be
> required to create KafkaPrincipal and if we want to maintain the old
> logging, we will have to create it as we do today.
> I will take care that we specify the Principal name in the log.
>
> Thanks again for all the reviews.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Feb 22, 2017 at 1:45 PM, Jun Rao  wrote:
>
> > Hi, Mayuresh,
> >
> > For logging the user name, we could do either way. We just need to make
> > sure the expected user name is logged. Also, currently, we are already
> > creating a KafkaPrincipal on every request. +1 on the latest KIP.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the comments.
> > >
> > > I will mention in the KIP : how this change doesn't affect the default
> > > authorizer implementation.
> > >
> > > Regarding, Currently, we log the principal name in the request log in
> > > RequestChannel, which has the format of "principalType + SEPARATOR +
> > > name;".
> > > It would be good if we can keep the same convention after this KIP. One
> > way
> > > to do that is to convert java.security.Principal to KafkaPrincipal for
> > > logging the requests.
> > > --- > This would mean we have to create a new KafkaPrincipal on each
> > > request. Would it be OK to just specify the name of the principal.
> > > Is there any major reason, we don't want to change the logging format?
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > >
> > >
> > > On Mon, Feb 20, 2017 at 10:18 PM, Jun Rao  wrote:
> > >
> > > > Hi, Mayuresh,
> > > >
> > > > Thanks for the updated KIP. A couple of more comments.
> > > >
> > > > 1. Do we convert java.security.Principal to KafkaPrincipal for
> > > > authorization check in SimpleAclAuthorizer? If so, it would be useful
> > to
> > > > mention that in the wiki so that people can understand how this
> change
> > > > doesn't affect the default authorizer implementation.
> > > >
> > > > 2. Currently, we log the principal name in the request log in
> > > > RequestChannel, which has the format of "principalType + SEPARATOR +
> > > > name;".
> > > > It would be good if we can keep the same convention after this KIP.
> One
> > > way
> > > > to do that is to convert java.security.Principal to KafkaPrincipal
> for
> > > > logging the requests.
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Fri, Feb 17, 2017 at 5:35 PM, Mayuresh Gharat <
> > > > gharatmayures...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > I have updated the KIP. Would you mind taking another look?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > > On Fri, Feb 17, 2017 at 4:42 PM, Mayuresh Gharat <
> > > > > gharatmayures...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Sure sounds good to me.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 1:54 PM, Jun Rao 
> wrote:
> > > > > >
> > > > > >> Hi, Mani,
> > > > > >>
> > > > > >> Good point on using PrincipalBuilder for SASL. It seems that
> > > > > >> PrincipalBuilder already has access to Authenticator. So, we
> could
> > > > just
> > > > > >> enable that in SaslChannelBuilder. We probably could do that in
> a
> > > > > separate
> > > > > >> KIP?
> > > > > >>
> > > > > >> Hi, Mayuresh,
> > > > > >>
> > > > > >> If you don't think there is a concrete use case for using
> > > > > >> PrincipalBuilder in
> > > > > >> kafka-acls.sh, perhaps we could do the simpler approach for now?
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jun
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Feb 17, 2017 at 12:23 PM, Mayuresh Gharat <
> > > > > >> gharatmayures...@gmail.com> wrote:
> > > > > >>
> > > > > >> > @Manikumar,
> > > > > >> >
> > > > > >> > Can you give an example how you are planning to use
> > > > PrincipalBuilder?
> > > > > >> >
> > > > > >> > @Jun
> > > > > >> > Yes, that is right. To give a brief overview, we just extract
> > the
> > > > cert
> > > > > >> and
> > > > > >> > hand it over to a third party library for creating a
> Principal.
> > So
> > > > we
> > > > > >> > cannot create a Principal from just a string.
> > > > > >> > The main motive behind adding the PrincipalBuilder for
> > > kafk-acls.sh
> > > > > was
> > > > > >> > that someone else (who can generate a Principal from map of
> > > > propertie,
> > > > > >> >  for example) can use it.
> > > > > >> > As I said, Linkedin is fine with not making any changes to
> > > > > Kafka-acls.sh
> > > > > >> > for now. But we thought that it would be a good 

[GitHub] kafka-site pull request #47: More edits on 0.10.2 web docs after the release

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/47


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Prasanna Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879885#comment-15879885
 ] 

Prasanna Gautam commented on KAFKA-2729:


[~junrao] Thanks for looking into this. Do you mind elaborating on what you 
need to change in the ZK api and whether 
https://issues.apache.org/jira/browse/KAFKA-3083 is going to solve it. The 
issue here is that -- in case of network partitions which are unrelated to the 
3 points and can happen at any time, this can happen and leave the brokers in 
messed up state until a restart. Can this be fixed by handling the ZK 
Connection errors?

If restarting the broker is the only fix, maybe the proper thing to do is to 
crash and let supervisor, etc. restart the service?

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Fwd: Producer acks=1, clean broker shutdown and data loss

[Nick Travers]

I didn't get any hits on this on users@, so I'm forwarding this to dev@.
Any explanation for this would be greatly appreciated!

-- Forwarded message --
From: Nick Travers 
Date: Sat, Feb 18, 2017 at 5:04 PM
Subject: Producer acks=1, clean broker shutdown and data loss
To: us...@kafka.apache.org


Hi - I'm trying to understand the expected behavior of the scenario in
which I have a producer with `acks=1` (i.e. partition leader acks only) and
I cleanly shut down a broker (via `KafkaServer#shutdown`).

I am running my test scenario with three brokers (0.10.1.1), with a default
replication count and default partition count of three. The producer (also
0.10.1.1) is writing to a single topic.

When I shut down one broker, I observe that in some instances there is data
loss on the topic that the producer was sending to, around the instant
where the broker was shut down and the producer sees that a new leader for
one of the partitions has been re-elected.

Reading the documentation [0], I see the following for the `acks=1` config:

This will mean the leader will write the record to its local log but will
> respond without awaiting full acknowledgement from all followers. In this
> case should the leader fail immediately after acknowledging the record but
> before the followers have replicated it then the record will be lost.


While it is clear to me that in the case of a _failure_ of this broker that
messages will be lost (as they have not yet been replicated to a follower),
it isn't clear to me what the expected behavior is in the case where this
broker is told to _cleanly_ shut down.

I understand that having a setup where `replicas=3`,
`min.insync.replicas=2` and `acks=-1` is much safer (and I've verified this
works as expected for the same test scenario), I'd like to understand the
semantics of the `acks=1` case nonetheless.

Thanks in advance.
- nick

[0]: https://github.com/apache/kafka/blob/0.10.1.1/clients/
src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L86

[jira] [Commented] (KAFKA-4762) Consumer throwing RecordTooLargeException even when messages are not that large

[Huadong Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879868#comment-15879868
 ] 

Huadong Liu commented on KAFKA-4762:


[~huxi_2b] You are right. [~neoeahit] confirmed that we did gzip compression.

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol 
says:
"As an optimization the server is allowed to return a partial message at the 
end of the message set. Clients should handle this case."

I guess that's why the partial compressed message is fetched by the consumer.

> Consumer throwing RecordTooLargeException even when messages are not that 
> large
> ---
>
> Key: KAFKA-4762
> URL: https://issues.apache.org/jira/browse/KAFKA-4762
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Vipul Singh
>
> We were just recently hit by a weird error. 
> Before going in any further, explaining of our service setup. we have a 
> producer which produces messages not larger than 256 kb of messages( we have 
> an explicit check about this on the producer side) and on the client side we 
> have a fetch limit of 512kb(max.partition.fetch.bytes is set to 524288 bytes) 
> Recently our client started to see this error:
> {quote}
> org.apache.kafka.common.errors.RecordTooLargeException: There are some 
> messages at [Partition=Offset]: {topic_name-0=9925056036} whose size is 
> larger than the fetch size 524288 and hence cannot be ever returned. Increase 
> the fetch size, or decrease the maximum message size the broker will allow.
> {quote}
> We tried consuming messages with another consumer, without any 
> max.partition.fetch.bytes limit, and it consumed fine. The messages were 
> small, and did not seem to be greater than 256 kb
> We took a log dump, and the log size looked fine.
> {quote}
> mpresscodec: NoCompressionCodec crc: 2473548911 keysize: 8
> offset: 9925056032 position: 191380053 isvalid: true payloadsize: 539 magic: 
> 0 compresscodec: NoCompressionCodec crc: 1656420267 keysize: 8
> offset: 9925056033 position: 191380053 isvalid: true payloadsize: 1551 magic: 
> 0 compresscodec: NoCompressionCodec crc: 2398479758 keysize: 8
> offset: 9925056034 position: 191380053 isvalid: true payloadsize: 1307 magic: 
> 0 compresscodec: NoCompressionCodec crc: 2845554215 keysize: 8
> offset: 9925056035 position: 191380053 isvalid: true payloadsize: 1520 magic: 
> 0 compresscodec: NoCompressionCodec crc: 3106984195 keysize: 8
> offset: 9925056036 position: 191713371 isvalid: true payloadsize: 1207 magic: 
> 0 compresscodec: NoCompressionCodec crc: 3462154435 keysize: 8
> offset: 9925056037 position: 191713371 isvalid: true payloadsize: 418 magic: 
> 0 compresscodec: NoCompressionCodec crc: 1536701802 keysize: 8
> offset: 9925056038 position: 191713371 isvalid: true payloadsize: 299 magic: 
> 0 compresscodec: NoCompressionCodec crc: 4112567543 keysize: 8
> offset: 9925056039 position: 191713371 isvalid: true payloadsize: 1571 magic: 
> 0 compresscodec: NoCompressionCodec crc: 3696994307 keysize: 8
> {quote}
> Has anyone seen something similar? or any points to troubleshoot this further
> Please Note: To overcome this issue, we deployed a new consumer, without this 
> limit of max.partition.fetch.bytes, and it worked fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Jay Kreps]

Hey Becket,

I get the problem we want to solve with this, but I don't think this is
something that makes sense as a user controlled knob that everyone sending
data to kafka has to think about. It is basically a bug, right?

First, as a technical question is it true that using the uncompressed size
for batching actually guarantees that you observe the limit? I think that
implies that compression always makes the messages smaller, which i think
usually true but is not guaranteed, right? e.g. if someone encrypts their
data which tends to randomize it and then enables compressesion, it could
slightly get bigger?

I also wonder if the rejected alternatives you describe couldn't be made to
work: basically try to be a bit better at estimation and recover when we
guess wrong. I don't think the memory usage should be a problem: isn't it
the same memory usage the consumer of that topic would need? And can't you
do the splitting and recompression in a streaming fashion? If we an make
the estimation rate low and the recovery cost is just ~2x the normal cost
for that batch that should be totally fine, right? (It's technically true
you might have to split more than once, but since you halve it each time I
think should you get a number of halvings that is logarithmic in the miss
size, which, with better estimation you'd hope would be super duper small).

Alternatively maybe we could work on the other side of the problem and try
to make it so that a small miss on message size isn't a big problem. I
think original issue was that max size and fetch size were tightly coupled
and the way memory in the consumer worked you really wanted fetch size to
be as small as possible because you'd use that much memory per fetched
partition and the consumer would get stuck if its fetch size wasn't big
enough. I think we made some progress on that issue and maybe more could be
done there so that a small bit of fuzziness around the size would not be an
issue?

-Jay



On Tue, Feb 21, 2017 at 12:30 PM, Becket Qin  wrote:

> Hi folks,
>
> I would like to start the discussion thread on KIP-126. The KIP propose
> adding a new configuration to KafkaProducer to allow batching based on
> uncompressed message size.
>
> Comments are welcome.
>
> The KIP wiki is following:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 126+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

Re: [DISCUSS] KIP-124: Request rate quotas

[Dong Lin]

Hey Rajini,

I think it makes a lot of sense to use io_thread_units as metric to quota
user's traffic here. LGTM overall. I have some questions regarding sensors.

- Can you be more specific in the KIP what sensors will be added? For
example, it will be useful to specify the name and attributes of these new
sensors.

- We currently have throttle-time and queue-size for byte-rate based quota.
Are you going to have separate throttle-time and queue-size for requests
throttled by io_thread_unit-based quota, or will they share the same sensor?

- Does the throttle-time in the ProduceResponse and FetchResponse contains
time due to io_thread_unit-based quota?

- Currently kafka server doesn't not provide any log or metrics that tells
whether any given clientId (or user) is throttled. This is not too bad
because we can still check the client-side byte-rate metric to validate
whether a given client is throttled. But with this io_thread_unit, there
will be no way to validate whether a given client is slow because it has
exceeded its io_thread_unit limit. It is necessary for user to be able to
know this information to figure how whether they have reached there quota
limit. How about we add log4j log on the server side to periodically print
the (client_id, byte-rate-throttle-time, io-thread-unit-throttle-time) so
that kafka administrator can figure those users that have reached their
limit and act accordingly?

Thanks,
Dong





On Wed, Feb 22, 2017 at 4:46 PM, Guozhang Wang  wrote:

> Made a pass over the doc, overall LGTM except a minor comment on the
> throttling implementation:
>
> Stated as "Request processing time throttling will be applied on top if
> necessary." I thought that it meant the request processing time throttling
> is applied first, but continue reading I found it actually meant to apply
> produce / fetch byte rate throttling first.
>
> Also the last sentence "The remaining delay if any is applied to the
> response." is a bit confusing to me. Maybe rewording it a bit?
>
>
> Guozhang
>
>
> On Wed, Feb 22, 2017 at 3:24 PM, Jun Rao  wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the updated KIP. The latest proposal looks good to me.
> >
> > Jun
> >
> > On Wed, Feb 22, 2017 at 2:19 PM, Rajini Sivaram  >
> > wrote:
> >
> > > Jun/Roger,
> > >
> > > Thank you for the feedback.
> > >
> > > 1. I have updated the KIP to use absolute units instead of percentage.
> > The
> > > property is called* io_thread_units* to align with the thread count
> > > property *num.io.threads*. When we implement network thread utilization
> > > quotas, we can add another property *network_thread_units.*
> > >
> > > 2. ControlledShutdown is already listed under the exempt requests. Jun,
> > did
> > > you mean a different request that needs to be added? The four requests
> > > currently exempt in the KIP are StopReplica, ControlledShutdown,
> > > LeaderAndIsr and UpdateMetadata. These are controlled using
> ClusterAction
> > > ACL, so it is easy to exclude and only throttle if unauthorized. I
> wasn't
> > > sure if there are other requests used only for inter-broker that needed
> > to
> > > be excluded.
> > >
> > > 3. I was thinking the smallest change would be to replace all
> references
> > to
> > > *requestChannel.sendResponse()* with a local method
> > > *sendResponseMaybeThrottle()* that does the throttling if any plus send
> > > response. If we throttle first in *KafkaApis.handle()*, the time spent
> > > within the method handling the request will not be recorded or used in
> > > throttling. We can look into this again when the PR is ready for
> review.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > >
> > > On Wed, Feb 22, 2017 at 5:55 PM, Roger Hoover 
> > > wrote:
> > >
> > > > Great to see this KIP and the excellent discussion.
> > > >
> > > > To me, Jun's suggestion makes sense.  If my application is allocated
> 1
> > > > request handler unit, then it's as if I have a Kafka broker with a
> > single
> > > > request handler thread dedicated to me.  That's the most I can use,
> at
> > > > least.  That allocation doesn't change even if an admin later
> increases
> > > the
> > > > size of the request thread pool on the broker.  It's similar to the
> CPU
> > > > abstraction that VMs and containers get from hypervisors or OS
> > > schedulers.
> > > > While different client access patterns can use wildly different
> amounts
> > > of
> > > > request thread resources per request, a given application will
> > generally
> > > > have a stable access pattern and can figure out empirically how many
> > > > "request thread units" it needs to meet it's throughput/latency
> goals.
> > > >
> > > > Cheers,
> > > >
> > > > Roger
> > > >
> > > > On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao  wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > Thanks for the updated KIP. A few more comments.
> > > > >
> > > > > 1. A concern of 

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Dong Lin]

Hey Becket,

I realized that Apurva has already raised similar questions. I think you
answered his question by saying that the request size will not be small. I
agree that there will be no impact on throughput if we can reach request
size limit with compression estimation disabled. But I am not sure that
will be the case. This is at least a concern where MM is mirroring traffic
for only a few partitions of high byte-in rate. Thus I am wondering if we
should do the optimization proposed above.

Thanks,
Dong

On Wed, Feb 22, 2017 at 6:39 PM, Dong Lin  wrote:

> Hey Becket,
>
> Thanks for the KIP. I have one question here.
>
> Suppose producer's batch.size=100 KB, max.in.flight.requests.per.connection=1.
> Since each ProduceRequest contains one batch per partition, it means that
> 100 KB compressed data will be produced per partition per round-trip time
> as of current implementation. If we disable compression estimation with
> this KIP, then produce can only produce 100 KB uncompressed data per
> partition per round-trip time. Suppose average compression ratio is 10,
> then there will be 10X difference in the bytes that are transmitted per
> round-trip time. The impact on the throughput can be big if mirror maker is
> producer to a remote cluster, even though the compression ratio may be the
> same.
>
> Given this observation, we should probably note in the KIP that user
> should bump up the producer's batch.size to the message.max.bytes
> configured on the broker, which by default is roughly 1MB, to achieve
> maximum possible throughput when compression estimation is disabled.
>
> Still, this can impact throughput of producer or MM that are producing
> highly compressible data. I think we can get around this problem by
> allowing each request to have multiple batches per partition as long as the
> size of these batches <= producer's batch.size config. Do you think it is
> worth doing?
>
> Thanks,
> Dong
>
>
>
> On Tue, Feb 21, 2017 at 7:56 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
>> Apurva has a point that can be documented for this config.
>>
>> Overall, LGTM +1.
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Tue, Feb 21, 2017 at 6:41 PM, Becket Qin  wrote:
>>
>> > Hi Apurva,
>> >
>> > Yes, it is true that the request size might be much smaller if the
>> batching
>> > is based on uncompressed size. I will let the users know about this.
>> That
>> > said, in practice, this is probably fine. For example, at LinkedIn, our
>> max
>> > message size is 1 MB, typically the compressed size would be 100 KB or
>> > larger, given that in most cases, there are many partitions, the request
>> > size would not be too small (typically around a few MB).
>> >
>> > At LinkedIn we do have some topics has various compression ratio. Those
>> are
>> > usually topics shared by different services so the data may differ a lot
>> > although they are in the same topic and similar fields.
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> > On Tue, Feb 21, 2017 at 6:17 PM, Apurva Mehta 
>> wrote:
>> >
>> > > Hi Becket, Thanks for the kip.
>> > >
>> > > I think one of the risks here is that when compression estimation is
>> > > disabled, you could have much smaller batches than expected, and
>> > throughput
>> > > could be hurt. It would be worth adding this to the documentation of
>> this
>> > > setting.
>> > >
>> > > Also, one of the rejected alternatives states that per topic
>> estimations
>> > > would not work when the compression of individual messages is
>> variable.
>> > > This is true in theory, but in practice one would expect Kafka topics
>> to
>> > > have fairly homogenous data, and hence should compress evenly. I was
>> > > curious if you have data which shows otherwise.
>> > >
>> > > Thanks,
>> > > Apurva
>> > >
>> > > On Tue, Feb 21, 2017 at 12:30 PM, Becket Qin 
>> > wrote:
>> > >
>> > > > Hi folks,
>> > > >
>> > > > I would like to start the discussion thread on KIP-126. The KIP
>> propose
>> > > > adding a new configuration to KafkaProducer to allow batching based
>> on
>> > > > uncompressed message size.
>> > > >
>> > > > Comments are welcome.
>> > > >
>> > > > The KIP wiki is following:
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > 126+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jiangjie (Becket) Qin
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
>
>

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Dong Lin]

Hey Becket,

Thanks for the KIP. I have one question here.

Suppose producer's batch.size=100 KB,
max.in.flight.requests.per.connection=1. Since each ProduceRequest contains
one batch per partition, it means that 100 KB compressed data will be
produced per partition per round-trip time as of current implementation. If
we disable compression estimation with this KIP, then produce can only
produce 100 KB uncompressed data per partition per round-trip time. Suppose
average compression ratio is 10, then there will be 10X difference in the
bytes that are transmitted per round-trip time. The impact on the
throughput can be big if mirror maker is producer to a remote cluster, even
though the compression ratio may be the same.

Given this observation, we should probably note in the KIP that user should
bump up the producer's batch.size to the message.max.bytes configured on
the broker, which by default is roughly 1MB, to achieve maximum possible
throughput when compression estimation is disabled.

Still, this can impact throughput of producer or MM that are producing
highly compressible data. I think we can get around this problem by
allowing each request to have multiple batches per partition as long as the
size of these batches <= producer's batch.size config. Do you think it is
worth doing?

Thanks,
Dong



On Tue, Feb 21, 2017 at 7:56 PM, Mayuresh Gharat  wrote:

> Apurva has a point that can be documented for this config.
>
> Overall, LGTM +1.
>
> Thanks,
>
> Mayuresh
>
> On Tue, Feb 21, 2017 at 6:41 PM, Becket Qin  wrote:
>
> > Hi Apurva,
> >
> > Yes, it is true that the request size might be much smaller if the
> batching
> > is based on uncompressed size. I will let the users know about this. That
> > said, in practice, this is probably fine. For example, at LinkedIn, our
> max
> > message size is 1 MB, typically the compressed size would be 100 KB or
> > larger, given that in most cases, there are many partitions, the request
> > size would not be too small (typically around a few MB).
> >
> > At LinkedIn we do have some topics has various compression ratio. Those
> are
> > usually topics shared by different services so the data may differ a lot
> > although they are in the same topic and similar fields.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Tue, Feb 21, 2017 at 6:17 PM, Apurva Mehta 
> wrote:
> >
> > > Hi Becket, Thanks for the kip.
> > >
> > > I think one of the risks here is that when compression estimation is
> > > disabled, you could have much smaller batches than expected, and
> > throughput
> > > could be hurt. It would be worth adding this to the documentation of
> this
> > > setting.
> > >
> > > Also, one of the rejected alternatives states that per topic
> estimations
> > > would not work when the compression of individual messages is variable.
> > > This is true in theory, but in practice one would expect Kafka topics
> to
> > > have fairly homogenous data, and hence should compress evenly. I was
> > > curious if you have data which shows otherwise.
> > >
> > > Thanks,
> > > Apurva
> > >
> > > On Tue, Feb 21, 2017 at 12:30 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I would like to start the discussion thread on KIP-126. The KIP
> propose
> > > > adding a new configuration to KafkaProducer to allow batching based
> on
> > > > uncompressed message size.
> > > >
> > > > Comments are welcome.
> > > >
> > > > The KIP wiki is following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 126+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

[GitHub] kafka-site issue #47: More edits on 0.10.2 web docs after the release

[junrao]

Github user junrao commented on the issue:

https://github.com/apache/kafka-site/pull/47
  
@guozhangwang : Thanks for the patch. LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879700#comment-15879700
 ] 

Jun Rao commented on KAFKA-2729:


Sorry to hear about the impact to production. Grant mentioned ZK session 
expiration, which is indeed a potential cause of this issue. A related issue 
has been reported in KAFKA-3083. The issue is that when the controller's ZK 
session expires and loses its controller-ship, it's possible for this zombie 
controller to continue updating ZK and/or sending LeaderAndIsrRequests to the 
brokers for a short period of time. When this happens, the broker may not have 
the most up-to-date information about leader and isr, which can lead to 
subsequent ZK failure when isr needs to be updated.

Fixing this issue requires us change the way how we use the ZK api and may take 
some time. In the interim, one suggestion is to make sure ZK session expiration 
never happens. This can be achieved by making sure that (1) ZK servers are 
performing, (2) the brokers don't have long GCs, (3) the ZK session expiration 
time is large enough.

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka-site issue #47: More edits on 0.10.2 web docs after the release

[derrickdoo]

Github user derrickdoo commented on the issue:

https://github.com/apache/kafka-site/pull/47
  
👍 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4753) KafkaConsumer susceptible to FetchResponse starvation

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879636#comment-15879636
 ] 

Jason Gustafson commented on KAFKA-4753:


[~onurkaraman] I agree in general. The user-driven IO model has been a source 
of trouble, and we have been gradually abandoning it. That said, I think it 
might still make sense to do the short-term mitigation first. It's a trivial 
change and helps with the current problem.

> KafkaConsumer susceptible to FetchResponse starvation
> -
>
> Key: KAFKA-4753
> URL: https://issues.apache.org/jira/browse/KAFKA-4753
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>
> FetchResponse starvation here means that the KafkaConsumer repeatedly fails 
> to fully form FetchResponses within the request timeout from a subset of the 
> brokers its fetching from while FetchResponses from the other brokers can get 
> fully formed and processed by the application.
> In other words, this ticket is concerned with scenarios where fetching from 
> some brokers hurts the progress of fetching from other brokers to the point 
> of repeatedly hitting a request timeout.
> Some FetchResponse starvation scenarios:
> 1. partition leadership of the consumer's assigned partitions is skewed 
> across brokers, causing uneven FetchResponse sizes across brokers.
> 2. the consumer seeks back on partitions on some brokers but not others, 
> causing uneven FetchResponse sizes across brokers.
> 3. the consumer's ability to keep up with various partitions across brokers 
> is skewed, causing uneven FetchResponse sizes across brokers.
> I've personally seen scenario 1 happen this past week to one of our users in 
> prod. They manually assigned partitions such that a few brokers led most of 
> the partitions while other brokers only led a single partition. When 
> NetworkClient sends out FetchRequests to different brokers in parallel with 
> an uneven partition distribution, FetchResponses from brokers who lead more 
> partitions will contain more data than FetchResponses from brokers who lead 
> few partitions. This means the small FetchResponses will get fully formed 
> quicker than larger FetchResponses. When the application eventually consumes 
> a smaller fully formed FetchResponses, the NetworkClient will send out a new 
> FetchRequest to the lightly-loaded broker. Their response will again come 
> back quickly while only marginal progress has been made on the larger 
> FetchResponse. Repeat this process several times and your application will 
> have potentially processed many smaller FetchResponses while the larger 
> FetchResponse made minimal progress and is forced to timeout, causing the 
> large FetchResponse to start all over again, which causes starvation.
> To mitigate the problem for the short term, I've suggested to our user that 
> they either:
> 1. bump up their "receive.buffer.bytes" beyond the current default of 64 KB 
> to something like 1 MB. This is the solution I short-term solution I 
> suggested they go with.
> 2. reduce the "max.partition.fetch.bytes" down from the current default of 1 
> MB to something like 100 KB. This solution wasn't advised as it could impact 
> broker performance.
> 3. ask our SRE's to run a partition reassignment to balance out the partition 
> leadership (partitions were already being led by their preferred leaders).
> 4. bump up their request timeout. It was set to open-source's former default 
> of 40 seconds.
> Contributing factors:
> 1. uneven FetchResponse sizes across brokers.
> 2. processing time of the polled ConsumerRecords.
> 3. "max.poll.records" increases the number of polls needed to consume a 
> FetchResponse, making constant-time overhead per poll magnified.
> 4. "max.poll.records" makes KafkaConsumer.poll bypass calls to 
> ConsumerNetworkClient.poll.
> 5. java.nio.channels.Selector.select, Selector.poll, NetworkClient.poll, and 
> ConsumerNetworkClient.poll can return before the poll timeout as soon as a 
> single channel is selected.
> 6. NetworkClient.poll is solely driven by the user thread with manual 
> partition assignment.
> So far I've only locally reproduced starvation scenario 1 and haven't even 
> attempted the other scenarios. Preventing the bypass of 
> ConsumerNetworkClient.poll (contributing factor 3) mitigates the issue, but 
> it seems starvation would still be possible.
> How to reproduce starvation scenario 1:
> 1. startup zookeeper
> 2. startup two brokers
> 3. create a topic t0 with two partitions led by broker 0 and create a topic 
> t1 with a single partition led by broker 1
> {code}
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic t0 
> > --replica-assignment 0,0
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic t1 
> > 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Im happy to compromise to keep it mutable but move to an append style api. (as 
in guava interables concat)

class Headers {
   Headers append(Iterable headers);
}


I don’t think we’d want prepend, this would give the idea of guaranteed 
ordering, when in actual fact we don’t provide that guarantee (.e.g one client 
can put headerA, then headerB, but another could put headerB then headerA, this 
shouldn’t cause issues), Also what if we changed to a hashmap for the internal 
implementation, its just a bucket of entries no ordering. I think we just need 
to provide an api to add/append headers.

This ok? If so ill update KIP to record this.

Cheers
Mike

On 23/02/2017, 00:37, "Jason Gustafson"  wrote:

The point about usability is fair. It's also reasonable to expect that
common use cases such as appending headers should be done efficiently.

Perhaps we could compromise with something like this?

class Headers {
 Headers append(Iterable headers);
 Headers prepend(Iterable headers);
}

That retains ease of use while still giving ourselves some flexibility in
the implementation.

-Jason


On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce 
wrote:

> I wasn’t referring to the headers needing to be copied, im meaning the
> fact we’d be forcing a new producer record to be created, with all the
> contents copied.
>
> i.e what will happen is utility method will be created or end up being
> used, which does this, and returns the new ProducerRecord instance.
>
> ProducerRecord  addHeader(ProducerRecord record, Header header){
> Return New ProducerRecord(record.key, record.value, record.timestamp…..,
> record.headers.concat(header))
> }
>
> To me this seems ugly, but will be inevitable if we don’t make adding
> headers to existing records a simple clean method call.
>
>
>
> On 22/02/2017, 22:57, "Michael Pearce"  wrote:
>
> Lazy init can achieve/avoid that.
>
> Re the concat, why don’t we implement that inside the Headers rather
> than causing everyone to implement this as adding headers in interceptors
> will be a dominant use case. We want a user friendly API. Having as a user
> having to code this instead of having the headers handle this for me seems
> redundant.
>
> On 22/02/2017, 22:34, "Jason Gustafson"  wrote:
>
> I thought the argument was against creating the extra objects
> unnecessarily
> (i.e. if they were not accessed). And note that making the Headers
> immutable doesn't necessarily mean that they need to be copied:
> you can do
> a trick like Guava's Iterables.concat to add additional headers
> without
> changing the underlying collections.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> michael.pea...@ig.com>
> wrote:
>
> > If the argument for not having a map holding the key, value
> pairs is due
> > to garbage creation of HashMap entry's, forcing the creation of
> a whole new
> > producer record to simply add a head, surely is creating a-lot
> more?
> > 
> > From: Jason Gustafson 
> > Sent: Wednesday, February 22, 2017 10:09 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The current producer interceptor API is this:
> >
> > ProducerRecord onSend(ProducerRecord record);
> >
> > So adding a header means creating a new ProducerRecord with a
> new header
> > added to the current headers and returning it. Would that not
> work?
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> michael.pea...@ig.com>
> > wrote:
> >
> > > So how would you have this work if not mutable where
> interceptors would
> > > add headers?
> > >
> > > Sent using OWA for iPhone
> > > 
> > > From: Jason Gustafson 
> > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > I think the point on the mutability of Headers is worth
> discussing a
> > little
> > > more. As far as I can tell, once the ProducerRecord (or
> ConsumerRecord)
> > is
> > > constructed, there should be no 

Re: [DISCUSS] KIP-124: Request rate quotas

[Guozhang Wang]

Made a pass over the doc, overall LGTM except a minor comment on the
throttling implementation:

Stated as "Request processing time throttling will be applied on top if
necessary." I thought that it meant the request processing time throttling
is applied first, but continue reading I found it actually meant to apply
produce / fetch byte rate throttling first.

Also the last sentence "The remaining delay if any is applied to the
response." is a bit confusing to me. Maybe rewording it a bit?


Guozhang


On Wed, Feb 22, 2017 at 3:24 PM, Jun Rao  wrote:

> Hi, Rajini,
>
> Thanks for the updated KIP. The latest proposal looks good to me.
>
> Jun
>
> On Wed, Feb 22, 2017 at 2:19 PM, Rajini Sivaram 
> wrote:
>
> > Jun/Roger,
> >
> > Thank you for the feedback.
> >
> > 1. I have updated the KIP to use absolute units instead of percentage.
> The
> > property is called* io_thread_units* to align with the thread count
> > property *num.io.threads*. When we implement network thread utilization
> > quotas, we can add another property *network_thread_units.*
> >
> > 2. ControlledShutdown is already listed under the exempt requests. Jun,
> did
> > you mean a different request that needs to be added? The four requests
> > currently exempt in the KIP are StopReplica, ControlledShutdown,
> > LeaderAndIsr and UpdateMetadata. These are controlled using ClusterAction
> > ACL, so it is easy to exclude and only throttle if unauthorized. I wasn't
> > sure if there are other requests used only for inter-broker that needed
> to
> > be excluded.
> >
> > 3. I was thinking the smallest change would be to replace all references
> to
> > *requestChannel.sendResponse()* with a local method
> > *sendResponseMaybeThrottle()* that does the throttling if any plus send
> > response. If we throttle first in *KafkaApis.handle()*, the time spent
> > within the method handling the request will not be recorded or used in
> > throttling. We can look into this again when the PR is ready for review.
> >
> > Regards,
> >
> > Rajini
> >
> >
> >
> > On Wed, Feb 22, 2017 at 5:55 PM, Roger Hoover 
> > wrote:
> >
> > > Great to see this KIP and the excellent discussion.
> > >
> > > To me, Jun's suggestion makes sense.  If my application is allocated 1
> > > request handler unit, then it's as if I have a Kafka broker with a
> single
> > > request handler thread dedicated to me.  That's the most I can use, at
> > > least.  That allocation doesn't change even if an admin later increases
> > the
> > > size of the request thread pool on the broker.  It's similar to the CPU
> > > abstraction that VMs and containers get from hypervisors or OS
> > schedulers.
> > > While different client access patterns can use wildly different amounts
> > of
> > > request thread resources per request, a given application will
> generally
> > > have a stable access pattern and can figure out empirically how many
> > > "request thread units" it needs to meet it's throughput/latency goals.
> > >
> > > Cheers,
> > >
> > > Roger
> > >
> > > On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao  wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > Thanks for the updated KIP. A few more comments.
> > > >
> > > > 1. A concern of request_time_percent is that it's not an absolute
> > value.
> > > > Let's say you give a user a 10% limit. If the admin doubles the
> number
> > of
> > > > request handler threads, that user now actually has twice the
> absolute
> > > > capacity. This may confuse people a bit. So, perhaps setting the
> quota
> > > > based on an absolute request thread unit is better.
> > > >
> > > > 2. ControlledShutdownRequest is also an inter-broker request and
> needs
> > to
> > > > be excluded from throttling.
> > > >
> > > > 3. Implementation wise, I am wondering if it's simpler to apply the
> > > request
> > > > time throttling first in KafkaApis.handle(). Otherwise, we will need
> to
> > > add
> > > > the throttling logic in each type of request.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thank you for the review.
> > > > >
> > > > > I have reverted to the original KIP that throttles based on request
> > > > handler
> > > > > utilization. At the moment, it uses percentage, but I am happy to
> > > change
> > > > to
> > > > > a fraction (out of 1 instead of 100) if required. I have added the
> > > > examples
> > > > > from this discussion to the KIP. Also added a "Future Work" section
> > to
> > > > > address network thread utilization. The configuration is named
> > > > > "request_time_percent" with the expectation that it can also be
> used
> > as
> > > > the
> > > > > limit for network thread utilization when that is implemented, so
> > that
> > > > > users have to set only one config for the two and not have to worry
> > > about
> > > > > 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

The point about usability is fair. It's also reasonable to expect that
common use cases such as appending headers should be done efficiently.

Perhaps we could compromise with something like this?

class Headers {
 Headers append(Iterable headers);
 Headers prepend(Iterable headers);
}

That retains ease of use while still giving ourselves some flexibility in
the implementation.

-Jason


On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce 
wrote:

> I wasn’t referring to the headers needing to be copied, im meaning the
> fact we’d be forcing a new producer record to be created, with all the
> contents copied.
>
> i.e what will happen is utility method will be created or end up being
> used, which does this, and returns the new ProducerRecord instance.
>
> ProducerRecord  addHeader(ProducerRecord record, Header header){
> Return New ProducerRecord(record.key, record.value, record.timestamp…..,
> record.headers.concat(header))
> }
>
> To me this seems ugly, but will be inevitable if we don’t make adding
> headers to existing records a simple clean method call.
>
>
>
> On 22/02/2017, 22:57, "Michael Pearce"  wrote:
>
> Lazy init can achieve/avoid that.
>
> Re the concat, why don’t we implement that inside the Headers rather
> than causing everyone to implement this as adding headers in interceptors
> will be a dominant use case. We want a user friendly API. Having as a user
> having to code this instead of having the headers handle this for me seems
> redundant.
>
> On 22/02/2017, 22:34, "Jason Gustafson"  wrote:
>
> I thought the argument was against creating the extra objects
> unnecessarily
> (i.e. if they were not accessed). And note that making the Headers
> immutable doesn't necessarily mean that they need to be copied:
> you can do
> a trick like Guava's Iterables.concat to add additional headers
> without
> changing the underlying collections.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> michael.pea...@ig.com>
> wrote:
>
> > If the argument for not having a map holding the key, value
> pairs is due
> > to garbage creation of HashMap entry's, forcing the creation of
> a whole new
> > producer record to simply add a head, surely is creating a-lot
> more?
> > 
> > From: Jason Gustafson 
> > Sent: Wednesday, February 22, 2017 10:09 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The current producer interceptor API is this:
> >
> > ProducerRecord onSend(ProducerRecord record);
> >
> > So adding a header means creating a new ProducerRecord with a
> new header
> > added to the current headers and returning it. Would that not
> work?
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> michael.pea...@ig.com>
> > wrote:
> >
> > > So how would you have this work if not mutable where
> interceptors would
> > > add headers?
> > >
> > > Sent using OWA for iPhone
> > > 
> > > From: Jason Gustafson 
> > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > I think the point on the mutability of Headers is worth
> discussing a
> > little
> > > more. As far as I can tell, once the ProducerRecord (or
> ConsumerRecord)
> > is
> > > constructed, there should be no need to further change the
> headers. Is
> > that
> > > correct? If so, then why not enforce that that is the case
> through the
> > API?
> > > One problem with mutability it that it constrains the
> implementation of
> > > Headers. For example, if we were backing with a byte slice,
> would we
> > recopy
> > > the bytes if a header is added or would we maintain a satellite
> > collection
> > > of added records. Seems not great either way. If we really
> think
> > mutability
> > > is needed, perhaps we could add a method to headers to convert
> it to a
> > > mutable type (e.g. a Headers.toMap)?
> > >
> > > I'm also with Ismael about exposing Headers.get(). I thought
> it might
> > make
> > > sense to have a method like this instead:
> > >
> > > Iterable findMatching(Pattern pattern);
> > >
> > > This makes the (potential) need to scan the headers clear in
> the API. I'd
> > > also be fine exposing no getter at all. In general, Ï think
> it's good to
> > > 

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879561#comment-15879561
 ] 

Onur Karaman commented on KAFKA-1895:
-

I think it's worth defining the relation between the two problems mentioned 
earlier:
# no means of access to raw FetchResponse data
# lack of a separate IO thread

I think Problem 1 is more of a performance problem while Problem 2 is a 
performance and usability problem (KAFKA-4753 shows that this is can lead to 
starvation).

Addressing Problem 1 doesn't solve Problem 2.

Addressing Problem 2 partially solves Problem 1. With a solution to Problem 2, 
we have the potential to also do the decompression/deserialization in the 
separate IO thread, removing decompression-in-user-thread performance concerns. 
But this wouldn't address the decompression-then-recompression performance 
concerns in MirrorMaker or perhaps some stream processing use-cases.

I think we need to solve both problems.

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4753) KafkaConsumer susceptible to FetchResponse starvation

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879558#comment-15879558
 ] 

Onur Karaman commented on KAFKA-4753:
-

[~ijuma] I don't think mitigation is the way to go.

The two solutions I can think of are to either:
1. introduce some sort of fair scheduling logic that polls against all brokers 
but only returns records to the user by waiting in a round-robin manner for a 
completed FetchResponse from a given broker. This would damage performance 
since you're blocking the processing of fully formed FetchResponses on the 
slowest fetch.

2. do all IO in a separate thread. I think we've been gravitating towards this 
since the beginning.
- January 2015: In KAFKA-1760, KafkaConsumer implemented as being entirely 
single-threaded with user-driven fetches and heartbeats.
- July 2015: I sent out an 
[email|https://www.mail-archive.com/dev@kafka.apache.org/msg31447.html] stating 
concerns with this approach, including the user-driven fetches and heartbeats. 
Becket and I proposed a [rough design for the 
fix|https://cwiki.apache.org/confluence/display/KAFKA/New+consumer+API+change+proposal]
 back then. Some nice API tweaks were made as a result of the discussion but 
the user-driven fetches and heartbeats remained.
- October 2015: In KAFKA-2397 I added LeaveGroupRequest to reduce the 
turnaround from a controlled client shutdown and rebalancing the group.
- February 2016: In KIP-62 (KAFKA-3888) we added "max.poll.records" in KIP-41 
to mitigate the impact of record processing on the user-driven heartbeats.
- August 2016: We added a separate heartbeat thread, synchronization logic 
surrounding NetworkClient, and retained poll-based liveness by automatically 
sending out LeaveGroupRequest when "max.poll.interval.ms" wasn't honored.
- February 2017: Some of the simplest uses of KafkaConsumer with manual 
partition assignment suffers from starvation again because of the user-driven 
fetch design decision.

I think we should try to make solution 2 work. We can retain the poll-based 
liveness by continuing what we do today: automatic sending of LeaveGroupRequest 
when "max.poll.interval.ms" isn't honored. This fix could also help KAFKA-1895 
as well, as we can potentially push decompression/deserialization in the 
separate IO thread.

> KafkaConsumer susceptible to FetchResponse starvation
> -
>
> Key: KAFKA-4753
> URL: https://issues.apache.org/jira/browse/KAFKA-4753
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>
> FetchResponse starvation here means that the KafkaConsumer repeatedly fails 
> to fully form FetchResponses within the request timeout from a subset of the 
> brokers its fetching from while FetchResponses from the other brokers can get 
> fully formed and processed by the application.
> In other words, this ticket is concerned with scenarios where fetching from 
> some brokers hurts the progress of fetching from other brokers to the point 
> of repeatedly hitting a request timeout.
> Some FetchResponse starvation scenarios:
> 1. partition leadership of the consumer's assigned partitions is skewed 
> across brokers, causing uneven FetchResponse sizes across brokers.
> 2. the consumer seeks back on partitions on some brokers but not others, 
> causing uneven FetchResponse sizes across brokers.
> 3. the consumer's ability to keep up with various partitions across brokers 
> is skewed, causing uneven FetchResponse sizes across brokers.
> I've personally seen scenario 1 happen this past week to one of our users in 
> prod. They manually assigned partitions such that a few brokers led most of 
> the partitions while other brokers only led a single partition. When 
> NetworkClient sends out FetchRequests to different brokers in parallel with 
> an uneven partition distribution, FetchResponses from brokers who lead more 
> partitions will contain more data than FetchResponses from brokers who lead 
> few partitions. This means the small FetchResponses will get fully formed 
> quicker than larger FetchResponses. When the application eventually consumes 
> a smaller fully formed FetchResponses, the NetworkClient will send out a new 
> FetchRequest to the lightly-loaded broker. Their response will again come 
> back quickly while only marginal progress has been made on the larger 
> FetchResponse. Repeat this process several times and your application will 
> have potentially processed many smaller FetchResponses while the larger 
> FetchResponse made minimal progress and is forced to timeout, causing the 
> large FetchResponse to start all over again, which causes starvation.
> To mitigate the problem for the short term, I've suggested to our user that 
> they either:
> 1. bump up their "receive.buffer.bytes" beyond the current 

[jira] [Commented] (KAFKA-4674) Frequent ISR shrinking and expanding and disconnects among brokers

[mjuarez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879553#comment-15879553
 ] 

mjuarez commented on KAFKA-4674:


This is happening to us in one our clusters in prod today, across multiple 
topics, and basically on all nodes.  The most impacted topic/partitions seem to 
be the ones that have the highest volume, but every topic that has some 
activity is being impacted.

Unlike [~344277...@qq.com], we are not seeing any exceptions in the logs.  See 
attached broker log file.  Please let me know if you need more log files and/or 
details.

> Frequent ISR shrinking and expanding and disconnects among brokers
> --
>
> Key: KAFKA-4674
> URL: https://issues.apache.org/jira/browse/KAFKA-4674
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, core
>Affects Versions: 0.10.0.1
> Environment: OS: Redhat Linux 2.6.32-431.el6.x86_64
> JDK: 1.8.0_45
>Reporter: Kaiming Wan
> Attachments: controller.log.rar, kafkabroker.20170221.log.zip, 
> server.log.2017-01-11-14, zookeeper.out.2017-01-11.log
>
>
> We use a kafka cluster with 3 brokers in production environment. It works 
> well for several month. Recently, we get the UnderReplicatedPartitions>0 
> warning mail. When we check the log, we find that the partition is always 
> experience ISR shrinking and expanding. And the disconnection exception can 
> be found in controller's log.
> We also found some deviant output in zookeeper's log which point to a 
> consumer(using old API depends on zookeeper ) which has stopped its work with 
> many lags.
> Actually, it is not the first time we encounter this problem. When we 
> first met this problem, we also found the same phenomenon and the log output. 
> We solve the problem by deleting the consumer node info in zookeeper. Then 
> everything goes well.
> However, this time, after we deleting the consumer which already have 
> large lag, the frequent ISR shrinking and expanding didn't stop for a very 
> long time(serveral hours). Though, the issue didn't affect our consumer and 
> producer, we think it will make our cluster unstable. So at last, we solve 
> this problem by restart the controller broker.
> And now I wander what cause this problem. I check the source code and 
> only know poll timeout will cause disconnection and ISR shrinking. Is the 
> issue related to zookeeper because it will not hold too many metadata 
> modification and make the replication fetch thread take more time?
> I upload the log file in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Kane Kim (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879554#comment-15879554
 ] 

Kane Kim commented on KAFKA-2729:
-

In my opinion it doesn't matter what's causing it (in our case that was indeed 
lost packets to zookeeper), the culprit is that brokers will not recover by 
itself until rolling restart. This is real problem and has to be fixed.


> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4674) Frequent ISR shrinking and expanding and disconnects among brokers

[mjuarez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mjuarez updated KAFKA-4674:
---
Attachment: kafkabroker.20170221.log.zip

> Frequent ISR shrinking and expanding and disconnects among brokers
> --
>
> Key: KAFKA-4674
> URL: https://issues.apache.org/jira/browse/KAFKA-4674
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, core
>Affects Versions: 0.10.0.1
> Environment: OS: Redhat Linux 2.6.32-431.el6.x86_64
> JDK: 1.8.0_45
>Reporter: Kaiming Wan
> Attachments: controller.log.rar, kafkabroker.20170221.log.zip, 
> server.log.2017-01-11-14, zookeeper.out.2017-01-11.log
>
>
> We use a kafka cluster with 3 brokers in production environment. It works 
> well for several month. Recently, we get the UnderReplicatedPartitions>0 
> warning mail. When we check the log, we find that the partition is always 
> experience ISR shrinking and expanding. And the disconnection exception can 
> be found in controller's log.
> We also found some deviant output in zookeeper's log which point to a 
> consumer(using old API depends on zookeeper ) which has stopped its work with 
> many lags.
> Actually, it is not the first time we encounter this problem. When we 
> first met this problem, we also found the same phenomenon and the log output. 
> We solve the problem by deleting the consumer node info in zookeeper. Then 
> everything goes well.
> However, this time, after we deleting the consumer which already have 
> large lag, the frequent ISR shrinking and expanding didn't stop for a very 
> long time(serveral hours). Though, the issue didn't affect our consumer and 
> producer, we think it will make our cluster unstable. So at last, we solve 
> this problem by restart the controller broker.
> And now I wander what cause this problem. I check the source code and 
> only know poll timeout will cause disconnection and ISR shrinking. Is the 
> issue related to zookeeper because it will not hold too many metadata 
> modification and make the replication fetch thread take more time?
> I upload the log file in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879543#comment-15879543
 ] 

Armin Braun commented on KAFKA-1895:


{quote}
   I don't know, supporting two deserializers sounds a lot easier than 
supporting two poll() methods.
{quote}

Fair point, though this only holds if the deserializer is not able to reuse the 
deserialized object (admittedly if we don't go that far, yes then it's probably 
not so hard to provide that interface). At least with interfaces `Writable` or 
Avro, the throughput you can achieve will take quite a hit if you don't have 
the ability for object reuse here.

{quote}
something like what Netty's ByteBuf provides).
{quote}

Yes exactly, you need shared state in Buffers and Pointers/Lengths between the 
iterator and the consumer. But I think moving the pointers with `next()` and 
having knowledge of all valid offsets/lengths shared between iterator and 
consumer makes this a relatively straightforward problem that has been dealt 
with elsewhere already too.

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1293

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-4786; Wait for heartbeat thread to terminate in consumer close

--
[...truncated 1.35 MB...]
kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879483#comment-15879483
 ] 

Jason Gustafson commented on KAFKA-1895:


I don't know, supporting two deserializers sounds a lot easier than supporting 
two poll() methods ;). No new configurations, no new APIs, we just need an 
instanceof check to tell which one to use. There might be better ideas out 
there though.

Making the buffers read-only is necessary, but not sufficient since the 
consumer needs to know when it can reuse them. It seems like you would need a 
way for the user to increment a reference count so that the consumer knows not 
to reuse the memory (something like what Netty's ByteBuf provides).

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hey Jun,

Thanks much for the explanation. I have some questions about 21 but that is
less important than 20. 20 would require considerable change to the KIP and
probably requires weeks to discuss again. Thus I would like to be very sure
that we agree on the problems with the current design as you mentioned and
there is no foreseeable problem with the alternate design.

Please see below I detail response. To summarize my points, I couldn't
figure out any non-trival drawback of the current design as compared to the
alternative design; and I couldn't figure out a good way to store offline
replicas in the alternative design. Can you see if these points make sense?
Thanks in advance for your time!!


1) The alternative design requires slightly more dependency on ZK. While
both solutions store created replicas in the ZK, the alternative design
would also store offline replicas in ZK but the current design doesn't. Thus

2) I am not sure that we should store offline replicas in znode
/brokers/ids/[brokerId]/failed-log-directory/[directory]. We probably don't
want to expose log directory path in zookeeper based on the concept that we
should only store logical information (e.g. topic, brokerId) in zookeeper's
path name. More specifically, we probably don't want to rename path in
zookeeper simply because user renamed a log director. And we probably don't
want to read/write these znode just because user manually moved replicas
between log directories.

3) I couldn't find a good way to store offline replicas in ZK in the
alternative design. We can store this information one znode per-topic,
per-brokerId, or per-brokerId-topic. All these choices have their own
problems. If we store it in per-topic znode then multiple brokers may need
to read/write offline replicas in the same znode which is generally bad. If
we store it per-brokerId then we effectively limit the maximum number of
topic-partition that can be stored on a broker by the znode size limit.
This contradicts the idea to expand the single broker capacity by throwing
in more disks. If we store it per-brokerId-topic, then when controller
starts, it needs to read number of brokerId*topic znodes which may double
the overall znode reads during controller startup.

4) The alternative design is less efficient than the current design in case
of log directory failure. The alternative design requires extra znode reads
in order to read offline replicas from zk while the current design requires
only one pair of LeaderAndIsrRequest and LeaderAndIsrResponse. The extra
znode reads will be proportional to the number of topics on the broker if
we store offline replicas per-brokerId-topic.

5) While I agree that the failure reporting should be done where the
failure is originated, I think the current design is consistent with what
we are already doing. With the current design, broker will send
notification via zookeeper and controller will send LeaderAndIsrRequest to
broker. This is similar to how broker sends isr change notification and
controller read latest isr from broker. If we do want broker to report
failure directly to controller, we should probably have broker send RPC
directly to controller as it sends ControllerShutdownRequest. I can do this
as well.

6) I don't think the current design requires additional state management in
each of the existing state handling such as topic creation or controller
failover. All these existing logic should stay exactly the same except that
the controller should recognize offline replicas on the live broker instead
of assuming all replicas on live brokers are live. But this additional
change is required in both the current design and the alternate design.
Thus there should be no difference between current design and the alternate
design with respect to these existing state handling logic in controller.

7) While I agree that the current design requires additional complexity in
the controller in order to handle LeaderAndIsrResponse and potentially
change partition and replica state to offline in the sate machines, I think
such logic is necessary in a well-designed controller either with the
alternate design or even without JBOD. Controller should be able to handle
error (e.g. ClusterAuthorizationException) in LeaderAndIsrResponse and
every responses in general. For example, if the controller hasn't received
LeaderAndIsrResponse after a given period if time, it probably means the
broker has hang and the controller should consider this broker as offline
and re-elect leader from other brokers. This would actually fix some
problem we have seen before at LinkedIn, where broker hangs due to
RAID-controller failure. In other words, I think it is a good idea for
controller to handle response.

8) I am not sure that the additional state management to handle
LeaderAndIsrResponse causes new types of synchronization. It is true that
the logic is not handled ZK event handling
thread. But the existing ControllerShutdownRequest is also not handled by

[GitHub] kafka pull request #2580: MINOR: Move ProtoUtils methods to ApiKeys

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2580


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-124: Request rate quotas

[Jun Rao]

Hi, Rajini,

Thanks for the updated KIP. The latest proposal looks good to me.

Jun

On Wed, Feb 22, 2017 at 2:19 PM, Rajini Sivaram 
wrote:

> Jun/Roger,
>
> Thank you for the feedback.
>
> 1. I have updated the KIP to use absolute units instead of percentage. The
> property is called* io_thread_units* to align with the thread count
> property *num.io.threads*. When we implement network thread utilization
> quotas, we can add another property *network_thread_units.*
>
> 2. ControlledShutdown is already listed under the exempt requests. Jun, did
> you mean a different request that needs to be added? The four requests
> currently exempt in the KIP are StopReplica, ControlledShutdown,
> LeaderAndIsr and UpdateMetadata. These are controlled using ClusterAction
> ACL, so it is easy to exclude and only throttle if unauthorized. I wasn't
> sure if there are other requests used only for inter-broker that needed to
> be excluded.
>
> 3. I was thinking the smallest change would be to replace all references to
> *requestChannel.sendResponse()* with a local method
> *sendResponseMaybeThrottle()* that does the throttling if any plus send
> response. If we throttle first in *KafkaApis.handle()*, the time spent
> within the method handling the request will not be recorded or used in
> throttling. We can look into this again when the PR is ready for review.
>
> Regards,
>
> Rajini
>
>
>
> On Wed, Feb 22, 2017 at 5:55 PM, Roger Hoover 
> wrote:
>
> > Great to see this KIP and the excellent discussion.
> >
> > To me, Jun's suggestion makes sense.  If my application is allocated 1
> > request handler unit, then it's as if I have a Kafka broker with a single
> > request handler thread dedicated to me.  That's the most I can use, at
> > least.  That allocation doesn't change even if an admin later increases
> the
> > size of the request thread pool on the broker.  It's similar to the CPU
> > abstraction that VMs and containers get from hypervisors or OS
> schedulers.
> > While different client access patterns can use wildly different amounts
> of
> > request thread resources per request, a given application will generally
> > have a stable access pattern and can figure out empirically how many
> > "request thread units" it needs to meet it's throughput/latency goals.
> >
> > Cheers,
> >
> > Roger
> >
> > On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao  wrote:
> >
> > > Hi, Rajini,
> > >
> > > Thanks for the updated KIP. A few more comments.
> > >
> > > 1. A concern of request_time_percent is that it's not an absolute
> value.
> > > Let's say you give a user a 10% limit. If the admin doubles the number
> of
> > > request handler threads, that user now actually has twice the absolute
> > > capacity. This may confuse people a bit. So, perhaps setting the quota
> > > based on an absolute request thread unit is better.
> > >
> > > 2. ControlledShutdownRequest is also an inter-broker request and needs
> to
> > > be excluded from throttling.
> > >
> > > 3. Implementation wise, I am wondering if it's simpler to apply the
> > request
> > > time throttling first in KafkaApis.handle(). Otherwise, we will need to
> > add
> > > the throttling logic in each type of request.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Jun,
> > > >
> > > > Thank you for the review.
> > > >
> > > > I have reverted to the original KIP that throttles based on request
> > > handler
> > > > utilization. At the moment, it uses percentage, but I am happy to
> > change
> > > to
> > > > a fraction (out of 1 instead of 100) if required. I have added the
> > > examples
> > > > from this discussion to the KIP. Also added a "Future Work" section
> to
> > > > address network thread utilization. The configuration is named
> > > > "request_time_percent" with the expectation that it can also be used
> as
> > > the
> > > > limit for network thread utilization when that is implemented, so
> that
> > > > users have to set only one config for the two and not have to worry
> > about
> > > > the internal distribution of the work between the two thread pools in
> > > > Kafka.
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao  wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > Thanks for the proposal.
> > > > >
> > > > > The benefit of using the request processing time over the request
> > rate
> > > is
> > > > > exactly what people have said. I will just expand that a bit.
> > Consider
> > > > the
> > > > > following case. The producer sends a produce request with a 10MB
> > > message
> > > > > but compressed to 100KB with gzip. The decompression of the message
> > on
> > > > the
> > > > > broker could take 10-15 seconds, during which time, a request
> handler
> > > > > thread is completely blocked. In this case, 

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879445#comment-15879445
 ] 

Armin Braun commented on KAFKA-1895:


{quote}
Maybe a simpler way to achieve that would be to have a new Deserializer type 
which works with byte buffers instead of byte arrays?
{quote}

Having buffers here would be better than arrays and already allow for a lot of 
optimizations. The downside I see here is that this would not come with reuse 
of the deserialized object quite so naturally. Also then you start having to 
support two kinds of deserializers(which would create a lot of complication in 
the code, just to give users the same thing they'd have from the 
RawRecordIterator interface - the option to reuse the deserialized object).

I agree here to some degree though:

{code}
 would make the consumer more confusing (we've had a tough enough time 
explaining how the current API works).
{code}

Yes this would make it more confusing. But on the other hand, at least the 
existing API would not change. If you do this via the deserializers you could 
probably keep things a little simpler but also slower to the outside, but at a 
pretty high price in terms of added complexity in the codebase.
Also my argument for this (adding another function) not being so bad would be 
that the interface already is fairly complex. Adding this method with proper 
javadoc (imo) will not be the reason for anyone to be tipped towards not 
understanding it anymore, who would have understood it before. Admittedly not 
the best argument in the world, but I feel like it's a reasonable tradeoff if 
you take the size of the necessary change into account (or the added complexity 
of different deserializer interfaces)?

{quote}
it might not be a great idea to give users direct access to the underlying 
buffers.
{quote}

This I would just solve by returning readonly buffers with the proper limit and 
position for a record set? This means the user must do some bounds checking, 
but this you have in Hadoop's RawKeyValueIterator too and is not an issue in my 
opinion. The other option would be to wrap the buffers in say `DataInput` to 
make the interface safer at the cost of a slight overhead (and the fact that 
some users may rather work from buffers than from DataInput).

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879445#comment-15879445
 ] 

Armin Braun edited comment on KAFKA-1895 at 2/22/17 11:16 PM:
--

{quote}
Maybe a simpler way to achieve that would be to have a new Deserializer type 
which works with byte buffers instead of byte arrays?
{quote}

Having buffers here would be better than arrays and already allow for a lot of 
optimizations. The downside I see here is that this would not come with reuse 
of the deserialized object quite so naturally. Also then you start having to 
support two kinds of deserializers(which would create a lot of complication in 
the code, just to give users the same thing they'd have from the 
RawRecordIterator interface - the option to reuse the deserialized object).

I agree here to some degree though:

{quote}
 would make the consumer more confusing (we've had a tough enough time 
explaining how the current API works).
{quote}

Yes this would make it more confusing. But on the other hand, at least the 
existing API would not change. If you do this via the deserializers you could 
probably keep things a little simpler but also slower to the outside, but at a 
pretty high price in terms of added complexity in the codebase.
Also my argument for this (adding another function) not being so bad would be 
that the interface already is fairly complex. Adding this method with proper 
javadoc (imo) will not be the reason for anyone to be tipped towards not 
understanding it anymore, who would have understood it before. Admittedly not 
the best argument in the world, but I feel like it's a reasonable tradeoff if 
you take the size of the necessary change into account (or the added complexity 
of different deserializer interfaces)?

{quote}
it might not be a great idea to give users direct access to the underlying 
buffers.
{quote}

This I would just solve by returning readonly buffers with the proper limit and 
position for a record set? This means the user must do some bounds checking, 
but this you have in Hadoop's RawKeyValueIterator too and is not an issue in my 
opinion. The other option would be to wrap the buffers in say `DataInput` to 
make the interface safer at the cost of a slight overhead (and the fact that 
some users may rather work from buffers than from DataInput).


was (Author: original-brownbear):
{quote}
Maybe a simpler way to achieve that would be to have a new Deserializer type 
which works with byte buffers instead of byte arrays?
{quote}

Having buffers here would be better than arrays and already allow for a lot of 
optimizations. The downside I see here is that this would not come with reuse 
of the deserialized object quite so naturally. Also then you start having to 
support two kinds of deserializers(which would create a lot of complication in 
the code, just to give users the same thing they'd have from the 
RawRecordIterator interface - the option to reuse the deserialized object).

I agree here to some degree though:

{code}
 would make the consumer more confusing (we've had a tough enough time 
explaining how the current API works).
{code}

Yes this would make it more confusing. But on the other hand, at least the 
existing API would not change. If you do this via the deserializers you could 
probably keep things a little simpler but also slower to the outside, but at a 
pretty high price in terms of added complexity in the codebase.
Also my argument for this (adding another function) not being so bad would be 
that the interface already is fairly complex. Adding this method with proper 
javadoc (imo) will not be the reason for anyone to be tipped towards not 
understanding it anymore, who would have understood it before. Admittedly not 
the best argument in the world, but I feel like it's a reasonable tradeoff if 
you take the size of the necessary change into account (or the added complexity 
of different deserializer interfaces)?

{quote}
it might not be a great idea to give users direct access to the underlying 
buffers.
{quote}

This I would just solve by returning readonly buffers with the proper limit and 
position for a record set? This means the user must do some bounds checking, 
but this you have in Hadoop's RawKeyValueIterator too and is not an issue in my 
opinion. The other option would be to wrap the buffers in say `DataInput` to 
make the interface safer at the cost of a slight overhead (and the fact that 
some users may rather work from buffers than from DataInput).

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Stephane Maarek]

https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11

Am I missing something ?

On 23 February 2017 at 9:21:08 am, Gwen Shapira (g...@confluent.io) wrote:

I saw them in Maven yesterday?

On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
 wrote:
> Awesome thanks a lot! When should we expect the dependencies to be
released
> in Maven? (including 2.12 scala)
>
> On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
>
> Thanks for driving the release, Ewen.
>
> Jun
>
> On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava 

> wrote:
>
>> The Apache Kafka community is pleased to announce the release for Apache
>> Kafka 0.10.2.0. This is a feature release which includes the completion
>> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
>> requests merged.
>>
>> All of the changes in this release can be found in the release notes:
>> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
>>
>> Apache Kafka is a distributed streaming platform with four four core
>> APIs:
>>
>> ** The Producer API allows an application to publish a stream records to
>> one or more Kafka topics.
>>
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>>
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an
>> output
>> stream to one or more output topics, effectively transforming the input
>> streams to output streams.
>>
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might capture
>> every change to a table.three key capabilities:
>>
>>
>> With these APIs, Kafka can be used for two broad classes of application:
>>
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>>
>> ** Building real-time streaming applications that transform or react to
>> the
>> streams of data.
>>
>>
>> You can download the source release from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka-0.10.2.0-src.tgz
>>
>> and binary releases from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.11-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.10-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.12-0.10.2.0.tgz
>> (experimental 2.12 artifact)
>>
>> Thanks to the 101 contributors on this release!
>>
>> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
>> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
>> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
>> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
>> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
>> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
>> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
>> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
>> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
>> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
>> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
>> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
>> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
>> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
>> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
>> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
>> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
>> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
>> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
>> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
>> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
>> Yang Wei, yaojuncn, Yuto Kawamura
>>
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at
>> http://kafka.apache.org/
>>
>> Thanks,
>> Ewen
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Stephane Maarek]

Awesome thanks a lot! When should we expect the dependencies to be released
in Maven? (including 2.12 scala)

On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:

Thanks for driving the release, Ewen.

Jun

On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava 
wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.2.0. This is a feature release which includes the completion
> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> requests merged.
>
> All of the changes in this release can be found in the release notes:
> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
>
> Apache Kafka is a distributed streaming platform with four four core
> APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react to
> the
> streams of data.
>
>
> You can download the source release from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka-0.10.2.0-src.tgz
>
> and binary releases from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.11-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.10-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.12-0.10.2.0.tgz
> (experimental 2.12 artifact)
>
> Thanks to the 101 contributors on this release!
>
> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> Yang Wei, yaojuncn, Yuto Kawamura
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
> Thanks,
> Ewen
>

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

I wasn’t referring to the headers needing to be copied, im meaning the fact 
we’d be forcing a new producer record to be created, with all the contents 
copied.

i.e what will happen is utility method will be created or end up being used, 
which does this, and returns the new ProducerRecord instance.

ProducerRecord  addHeader(ProducerRecord record, Header header){
Return New ProducerRecord(record.key, record.value, record.timestamp….., 
record.headers.concat(header))
}

To me this seems ugly, but will be inevitable if we don’t make adding headers 
to existing records a simple clean method call.



On 22/02/2017, 22:57, "Michael Pearce"  wrote:

Lazy init can achieve/avoid that.

Re the concat, why don’t we implement that inside the Headers rather than 
causing everyone to implement this as adding headers in interceptors will be a 
dominant use case. We want a user friendly API. Having as a user having to code 
this instead of having the headers handle this for me seems redundant.

On 22/02/2017, 22:34, "Jason Gustafson"  wrote:

I thought the argument was against creating the extra objects 
unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can 
do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.

-Jason

On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce 
wrote:

> If the argument for not having a map holding the key, value pairs is 
due
> to garbage creation of HashMap entry's, forcing the creation of a 
whole new
> producer record to simply add a head, surely is creating a-lot more?
> 
> From: Jason Gustafson 
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord onSend(ProducerRecord record);
>
> So adding a header means creating a new ProducerRecord with a new 
header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce 

> wrote:
>
> > So how would you have this work if not mutable where interceptors 
would
> > add headers?
> >
> > Sent using OWA for iPhone
> > 
> > From: Jason Gustafson 
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or 
ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. 
Is
> that
> > correct? If so, then why not enforce that that is the case through 
the
> API?
> > One problem with mutability it that it constrains the 
implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it 
to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it 
might
> make
> > sense to have a method like this instead:
> >
> > Iterable findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the 
API. I'd
> > also be fine exposing no getter at all. In general, Ï think it's 
good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 

> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally 
are map
> > > styled interfaces and what I believe is most expected styled thus 
the
> > Key,
 

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879418#comment-15879418
 ] 

Jason Gustafson commented on KAFKA-1895:


It's an interesting idea, but it's also a big change to the API, right? Having 
two ways to poll the data and with differing semantics would make the consumer 
more confusing (we've had a tough enough time explaining how the current API 
works). I think we should be clear about the problem we are trying to solve. It 
sounds like what you are looking for is to avoid copying the message data. 
Maybe a simpler way to achieve that would be to have a new Deserializer type 
which works with byte buffers instead of byte arrays? One challenge is that the 
consumer would probably like to control its own memory pool (see 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Bound+Fetch+memory+usage+in+the+consumer),
 so it might not be a great idea to give users direct access to the underlying 
buffers. 

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Lazy init can achieve/avoid that.

Re the concat, why don’t we implement that inside the Headers rather than 
causing everyone to implement this as adding headers in interceptors will be a 
dominant use case. We want a user friendly API. Having as a user having to code 
this instead of having the headers handle this for me seems redundant.

On 22/02/2017, 22:34, "Jason Gustafson"  wrote:

I thought the argument was against creating the extra objects unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.

-Jason

On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce 
wrote:

> If the argument for not having a map holding the key, value pairs is due
> to garbage creation of HashMap entry's, forcing the creation of a whole 
new
> producer record to simply add a head, surely is creating a-lot more?
> 
> From: Jason Gustafson 
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord onSend(ProducerRecord record);
>
> So adding a header means creating a new ProducerRecord with a new header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce 
> wrote:
>
> > So how would you have this work if not mutable where interceptors would
> > add headers?
> >
> > Sent using OWA for iPhone
> > 
> > From: Jason Gustafson 
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. Is
> that
> > correct? If so, then why not enforce that that is the case through the
> API?
> > One problem with mutability it that it constrains the implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it might
> make
> > sense to have a method like this instead:
> >
> > Iterable findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the API. 
I'd
> > also be fine exposing no getter at all. In general, Ï think it's good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally are 
map
> > > styled interfaces and what I believe is most expected styled thus the
> > Key,
> > > Value setter.
> > > Also it would mean rather than an interface, we would be making our
> > > internal header impl object we have for the array, exposed. E.g. if we
> > had
> > > a Map really this would be Map.Entry interface, this is the same
> reasons
> > on
> > > the map interface I cannot actually make the underlying Node object
> > that’s
> > > the implementation for HashMap, so that internals can be changed.
> > >
> > >
> > > On point 3,
> > >
> > > I think it people do expect it to be performant, thus why originally
> > > concern I raised with using an array for to me is an early memory
> > > optimisation. I think the user experience of properties/headers is on 
a
> > > get/set model. This is why its important we have encapsulated logic
> that
> > > then allows us to change this to a map, if this becomes and issue, and
> > the
> > > memory overhead of hashmap is less so.
> > >
> > >
> > >
> > >
> > > On 22/02/2017, 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[James Cheng]

Woohoo! Thanks for running the release, Ewen!

-James

> On Feb 22, 2017, at 12:33 AM, Ewen Cheslack-Postava  wrote:
> 
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.2.0. This is a feature release which includes the completion
> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> requests merged.
> 
> All of the changes in this release can be found in the release notes:
> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> 
> Apache Kafka is a distributed streaming platform with four four core
> APIs:
> 
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
> 
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
> 
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
> 
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
> 
> 
> With these APIs, Kafka can be used for two broad classes of application:
> 
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
> 
> ** Building real-time streaming applications that transform or react to
> the
> streams of data.
> 
> 
> You can download the source release from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka-0.10.2.0-src.tgz
> 
> and binary releases from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
> (experimental 2.12 artifact)
> 
> Thanks to the 101 contributors on this release!
> 
> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> Yang Wei, yaojuncn, Yuto Kawamura
> 
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
> 
> Thanks,
> Ewen

[jira] [Comment Edited] (KAFKA-4787) KafkaStreams close() is not reentrant

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879400#comment-15879400
 ] 

Matthias J. Sax edited comment on KAFKA-4787 at 2/22/17 10:55 PM:
--

My bad. Did read carefully enough... However, it seems to be a duplicate: 
https://issues.apache.org/jira/browse/KAFKA-4366
You can apply a timeout to close to resolve the issue.

"Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
intentional) – but to make Kafka Streams robust it should handle any sequence 
of close() invocations in particular gracefully." -> as the call happens from 
two different threads, I am not sure if streams could do anything about it. 
Maybe, we should add a "best practice" notice to apply a timeout or not call 
{{close()}} from an exception handler...

On the other handed, I am wondering if we might be able to handle exception 
differently after we did enter {{close()}} to avoid calling the thread 
exception handler. Would it be sufficient to remove all registered exception 
handlers with {{close()}} before starting the actual shut down steps... ?


was (Author: mjsax):
My bad. Did read carefully enough... However, it seems to be a duplicate: 
https://issues.apache.org/jira/browse/KAFKA-4366
You can apply a timeout to close to resolve the issue.

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4787) KafkaStreams close() is not reentrant

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879400#comment-15879400
 ] 

Matthias J. Sax commented on KAFKA-4787:


My bad. Did read carefully enough... However, it seems to be a duplicate: 
https://issues.apache.org/jira/browse/KAFKA-4366
You can apply a timeout to close to resolve the issue.

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Ismael Juma]

The index is stale, the files are there:

http://central.maven.org/maven2/org/apache/kafka/kafka_2.11/0.10.2.0/
http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/0.10.2.0/

Ismael

On Wed, Feb 22, 2017 at 10:26 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11
>
> Am I missing something ?
>
> On 23 February 2017 at 9:21:08 am, Gwen Shapira (g...@confluent.io) wrote:
>
> I saw them in Maven yesterday?
>
> On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
>  wrote:
> > Awesome thanks a lot! When should we expect the dependencies to be
> released
> > in Maven? (including 2.12 scala)
> >
> > On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
> >
> > Thanks for driving the release, Ewen.
> >
> > Jun
> >
> > On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava <
> ewe...@apache.org>
>
> > wrote:
> >
> >> The Apache Kafka community is pleased to announce the release for Apache
> >> Kafka 0.10.2.0. This is a feature release which includes the completion
> >> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> >> requests merged.
> >>
> >> All of the changes in this release can be found in the release notes:
> >> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> >>
> >> Apache Kafka is a distributed streaming platform with four four core
> >> APIs:
> >>
> >> ** The Producer API allows an application to publish a stream records to
> >> one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output
> >> stream to one or more output topics, effectively transforming the input
> >> streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might capture
> >> every change to a table.three key capabilities:
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react to
> >> the
> >> streams of data.
> >>
> >>
> >> You can download the source release from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka-0.10.2.0-src.tgz
> >>
> >> and binary releases from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.11-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.10-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.12-0.10.2.0.tgz
> >> (experimental 2.12 artifact)
> >>
> >> Thanks to the 101 contributors on this release!
> >>
> >> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> >> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> >> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> >> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> >> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> >> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> >> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> >> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> >> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> >> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> >> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> >> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> >> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> >> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> >> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> >> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> >> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> >> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> >> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> >> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> >> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> >> Yang Wei, yaojuncn, Yuto Kawamura
> >>
> >> We welcome your help and feedback. For more information on how to
> >> report problems, and to get involved, visit the project website at
> >> http://kafka.apache.org/
> >>
> >> Thanks,
> >> Ewen
> >>
>
>
>
> --
> Gwen Shapira
> 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

I thought the argument was against creating the extra objects unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.

-Jason

On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce 
wrote:

> If the argument for not having a map holding the key, value pairs is due
> to garbage creation of HashMap entry's, forcing the creation of a whole new
> producer record to simply add a head, surely is creating a-lot more?
> 
> From: Jason Gustafson 
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord onSend(ProducerRecord record);
>
> So adding a header means creating a new ProducerRecord with a new header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce 
> wrote:
>
> > So how would you have this work if not mutable where interceptors would
> > add headers?
> >
> > Sent using OWA for iPhone
> > 
> > From: Jason Gustafson 
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. Is
> that
> > correct? If so, then why not enforce that that is the case through the
> API?
> > One problem with mutability it that it constrains the implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it might
> make
> > sense to have a method like this instead:
> >
> > Iterable findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the API. I'd
> > also be fine exposing no getter at all. In general, Ï think it's good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally are map
> > > styled interfaces and what I believe is most expected styled thus the
> > Key,
> > > Value setter.
> > > Also it would mean rather than an interface, we would be making our
> > > internal header impl object we have for the array, exposed. E.g. if we
> > had
> > > a Map really this would be Map.Entry interface, this is the same
> reasons
> > on
> > > the map interface I cannot actually make the underlying Node object
> > that’s
> > > the implementation for HashMap, so that internals can be changed.
> > >
> > >
> > > On point 3,
> > >
> > > I think it people do expect it to be performant, thus why originally
> > > concern I raised with using an array for to me is an early memory
> > > optimisation. I think the user experience of properties/headers is on a
> > > get/set model. This is why its important we have encapsulated logic
> that
> > > then allows us to change this to a map, if this becomes and issue, and
> > the
> > > memory overhead of hashmap is less so.
> > >
> > >
> > >
> > >
> > > On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" <
> > > isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:
> > >
> > > Hi all,
> > >
> > > Great to see the progress that has been achieved on this one. :) A
> > few
> > > comments regarding the APIs (I'm still reviewing the message format
> > > changes):
> > >
> > > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord`
> should
> > be
> > > named `headers` (we avoid the `get` prefix in Kafka)
> > >
> > > 2. The `Headers` class is mutable (there's an `add` method). Does
> it
> > > need
> > > to be? If so, it would be good to explain why. Related to that, we
> > > should
> > > also explain the thinking around thread-safety. If we keep the
> `add`
> > > method, it may make sense for it to take a `Header` (that way we
> can
> > > add
> > > things to 

[jira] [Comment Edited] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879170#comment-15879170
 ] 

Armin Braun edited comment on KAFKA-1895 at 2/22/17 10:30 PM:
--

[~hachikuji]

{quote}

How would you propose to expose RawRecordIterator in the consumer API? The 
problem is that IO is currently driven through the poll() API, so introducing 
something else would be a bit weird (though maybe there's a nice way to do it).

{quote}

I admit this is potentially not ideal in terms of nice API design but I can see 
a relatively short route to implementation when adding a method like below to 
`org.apache.kafka.clients.consumer.Consumer`:

{code}
public RawRecordIterator pollRaw(long timeout);
{code}
 
which would simply return the RawRecordIterator that you could also use to back 
the current:

{code}
/**
 * @see KafkaConsumer#poll(long)
 */
public ConsumerRecords poll(long timeout);
{code}

... then you could deserialize the records from the `RawRecordIterator`  when 
instantiating `ConsumerRecords` from the `RawRecordIterator`.
I would also just deserialize when instantiating `ConsumerRecords` from 
`RawRecordIterator` and have that drain the full `RawRecordIterator` to support:

{code}
/**
 * Get just the records for the given partition
 * 
 * @param partition The partition to get records for
 */
public List> records(TopicPartition partition) {
List> recs = this.records.get(partition);
if (recs == null)
return Collections.emptyList();
else
return Collections.unmodifiableList(recs);
}
{code}

still. So no API change needed here + I think moving to a reused set of 
`ByteBuffer` abstracted by `RawRecordIterator` instead of the current solution 
constantly allocating `ByteBuffer` wrapped in things like 
`org.apache.kafka.common.network.NetworkReceive` would be significantly faster 
by saving lots of system calls (wouldn't have to read the header(size) bytes 
separately anymore if pointers on the underlying buffers are used instead of 
setting up buffers and hence byte[] after making that extra system call just to 
read 4 bytes).

So in a nutshell, just allow `pollRaw` which would always return a 
`RawRecordIterator`. Retain all existing APIs and instantiate `ConsumerRecords` 
from the iterator in a (copying) way that keeps `ConsumerRecord` immutable.

Hope this was understandable :) I understand there are a few tricky spots here 
in the implementation (though given that a wealth of work on this exists 
especially in Spark it's not that hard in 2017 imo), but just from the API I 
think the approach is sound. Plus it doesn't require any trickery where 
`ConsumerRecord` instances either become mutable or must be "passed back 
somewhere" to make them reusable, while still offering a 0 GC raw API.

EDIT:

So the semantic I would be looking for would be this:

{code}
while(/*  whatever your use case desires */) { 
Rawrecorditerator rawIterator = consumer.pollRaw(); //same object/iterator 
should be returned here every time ... filled with new content.
   while(rawIterator.next()) {
  // do something with key and value buffer contents
   }
}
{code}


was (Author: original-brownbear):
[~hachikuji]

{quote}

How would you propose to expose RawRecordIterator in the consumer API? The 
problem is that IO is currently driven through the poll() API, so introducing 
something else would be a bit weird (though maybe there's a nice way to do it).

{quote}

I admit this is potentially not ideal in terms of nice API design but I can see 
a relatively short route to implementation when adding a method like below to 
`org.apache.kafka.clients.consumer.Consumer`:

{code}
public RawRecordIterator pollRaw(long timeout);
{code}
 
which would simply return the RawRecordIterator that you could also use to back 
the current:

{code}
/**
 * @see KafkaConsumer#poll(long)
 */
public ConsumerRecords poll(long timeout);
{code}

... then you could deserialize the records from the `RawRecordIterator`  when 
instantiating `ConsumerRecords` from the `RawRecordIterator`.
I would also just deserialize when instantiating `ConsumerRecords` from 
`RawRecordIterator` and have that drain the full `RawRecordIterator` to support:

{code}
/**
 * Get just the records for the given partition
 * 
 * @param partition The partition to get records for
 */
public List> records(TopicPartition partition) {
List> recs = this.records.get(partition);
if (recs == null)
return Collections.emptyList();
else
return Collections.unmodifiableList(recs);
}
{code}

still. So no API change needed here + I think moving to a reused set of 
`ByteBuffer` abstracted by 

[jira] [Comment Edited] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879170#comment-15879170
 ] 

Armin Braun edited comment on KAFKA-1895 at 2/22/17 10:29 PM:
--

[~hachikuji]

{quote}

How would you propose to expose RawRecordIterator in the consumer API? The 
problem is that IO is currently driven through the poll() API, so introducing 
something else would be a bit weird (though maybe there's a nice way to do it).

{quote}

I admit this is potentially not ideal in terms of nice API design but I can see 
a relatively short route to implementation when adding a method like below to 
`org.apache.kafka.clients.consumer.Consumer`:

{code}
public RawRecordIterator pollRaw(long timeout);
{code}
 
which would simply return the RawRecordIterator that you could also use to back 
the current:

{code}
/**
 * @see KafkaConsumer#poll(long)
 */
public ConsumerRecords poll(long timeout);
{code}

... then you could deserialize the records from the `RawRecordIterator`  when 
instantiating `ConsumerRecords` from the `RawRecordIterator`.
I would also just deserialize when instantiating `ConsumerRecords` from 
`RawRecordIterator` and have that drain the full `RawRecordIterator` to support:

{code}
/**
 * Get just the records for the given partition
 * 
 * @param partition The partition to get records for
 */
public List> records(TopicPartition partition) {
List> recs = this.records.get(partition);
if (recs == null)
return Collections.emptyList();
else
return Collections.unmodifiableList(recs);
}
{code}

still. So no API change needed here + I think moving to a reused set of 
`ByteBuffer` abstracted by `RawRecordIterator` instead of the current solution 
constantly allocating `ByteBuffer` wrapped in things like 
`org.apache.kafka.common.network.NetworkReceive` would be significantly faster 
by saving lots of system calls (wouldn't have to read the header(size) bytes 
separately anymore if pointers on the underlying buffers are used instead of 
setting up buffers and hence byte[] after making that extra system call just to 
read 4 bytes).

So in a nutshell, just allow `pollRaw` which would always return a 
`RawRecordIterator`. Retain all existing APIs and instantiate `ConsumerRecords` 
from the iterator in a (copying) way that keeps `ConsumerRecord` immutable.

Hope this was understandable :) I understand there are a few tricky spots here 
in the implementation (though given that a wealth of work on this exists 
especially in Spark it's not that hard in 2017 imo), but just from the API I 
think the approach is sound. Plus it doesn't require any trickery where 
`ConsumerRecord` instances either become mutable or must be "passed back 
somewhere" to make them reusable, while still offering a 0 GC raw API.

EDIT:

So the semantic I would be looking for would be this:

{code}
while(/*  whatever your use case desires */) { 
Rawrecorditerator rawIterator = consumer.pollRaw(); //same object/iterator 
should be returned here every time ... filled with new content.
   while(rawIterator.next()) {
  // do something with key and value buffer contents
   }
}


was (Author: original-brownbear):
[~hachikuji]

{quote}

How would you propose to expose RawRecordIterator in the consumer API? The 
problem is that IO is currently driven through the poll() API, so introducing 
something else would be a bit weird (though maybe there's a nice way to do it).

{quote}

I admit this is potentially not ideal in terms of nice API design but I can see 
a relatively short route to implementation when adding a method like below to 
`org.apache.kafka.clients.consumer.Consumer`:

{code}
public RawRecordIterator pollRaw(long timeout);
{code}
 
which would simply return the RawRecordIterator that you could also use to back 
the current:

{code}
/**
 * @see KafkaConsumer#poll(long)
 */
public ConsumerRecords poll(long timeout);
{code}

... then you could deserialize the records from the `RawRecordIterator`  when 
instantiating `ConsumerRecords` from the `RawRecordIterator`.
I would also just deserialize when instantiating `ConsumerRecords` from 
`RawRecordIterator` and have that drain the full `RawRecordIterator` to support:

{code}
/**
 * Get just the records for the given partition
 * 
 * @param partition The partition to get records for
 */
public List> records(TopicPartition partition) {
List> recs = this.records.get(partition);
if (recs == null)
return Collections.emptyList();
else
return Collections.unmodifiableList(recs);
}
{code}

still. So no API change needed here + I think moving to a reused set of 
`ByteBuffer` abstracted by `RawRecordIterator` 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Gwen Shapira]

I looked here:
https://repository.apache.org/#nexus-search;gav~org.apache.kafka~kafka_2.12kw,versionexpand


On Wed, Feb 22, 2017 at 2:26 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:

> https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11
>
> Am I missing something ?
>
> On 23 February 2017 at 9:21:08 am, Gwen Shapira (g...@confluent.io) wrote:
>
> I saw them in Maven yesterday?
>
> On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
>  wrote:
> > Awesome thanks a lot! When should we expect the dependencies to be
> released
> > in Maven? (including 2.12 scala)
> >
> > On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
> >
> > Thanks for driving the release, Ewen.
> >
> > Jun
> >
> > On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava <
> ewe...@apache.org>
> > wrote:
> >
> >> The Apache Kafka community is pleased to announce the release for
> Apache
> >> Kafka 0.10.2.0. This is a feature release which includes the completion
> >> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> >> requests merged.
> >>
> >> All of the changes in this release can be found in the release notes:
> >> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> >>
> >> Apache Kafka is a distributed streaming platform with four four core
> >> APIs:
> >>
> >> ** The Producer API allows an application to publish a stream records
> to
> >> one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output
> >> stream to one or more output topics, effectively transforming the input
> >> streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might
> capture
> >> every change to a table.three key capabilities:
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of
> application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react to
> >> the
> >> streams of data.
> >>
> >>
> >> You can download the source release from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka-0.10.2.0-src.tgz
> >>
> >> and binary releases from
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.11-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.10-0.10.2.0.tgz
> >> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> >> 0/kafka_2.12-0.10.2.0.tgz
> >> (experimental 2.12 artifact)
> >>
> >> Thanks to the 101 contributors on this release!
> >>
> >> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> >> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> >> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> >> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> >> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> >> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> >> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> >> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> >> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> >> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> >> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> >> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> >> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> >> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> >> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> >> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> >> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> >> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> >> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> >> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> >> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> >> Yang Wei, yaojuncn, Yuto Kawamura
> >>
> >> We welcome your help and feedback. For more information on how to
> >> report problems, and to get involved, visit the project website at
> >> http://kafka.apache.org/
> >>
> >> Thanks,
> >> Ewen
> >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 <(650)%20450-2760> | @gwenshap
> 

[jira] [Comment Edited] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874894#comment-15874894
 ] 

Armin Braun edited comment on KAFKA-1895 at 2/22/17 10:26 PM:
--

[~jkreps] Maybe let me give the perspective of a user, using Kafka in a 
situation where high throughput and a low GC footprint are key and this change 
would be extremely valuable.

What I would ideally like to see here and in KAFKA-2045 is this:

Create a lower level interface along the lines of Hadoop's 
[RawKeyValueIterator](http://atetric.com/atetric/javadoc/org.apache.hadoop/hadoop-mapreduce-client-core/2.6.2/src-html/org/apache/hadoop/mapred/RawKeyValueIterator.html)
 that would back the current KafkaConsumer interface:

{code}public interface RawRecordIterator {
/**
 * Gets the current raw key.
 *
 * @return Gets the current raw key
 * @throws IOException
 */
ByteBuffer getKey() throws IOException;

/**
 * Gets the current raw value.
 *
 * @return Gets the current raw value
 * @throws IOException
 */
ByteBuffer getValue() throws IOException;

/**
 * Sets up the current key and value (for getKey and getValue).
 *
 * @return true if there exists a key/value,
 * false otherwise.
 * false implies the need for making the next call to
 * poll()
 * @throws IOException
 */
boolean next() throws IOException;
}
{code}

I think this one is pretty idea if implemented properly:

* If you start from some initial size for two backing `ByteBuffer`s and grow 
them as needed, if a poll doesn't fit into the existing ones you can have a 
nice and fast zero copy iterator (simply store the offsets and lengths for each 
record in `int` arrays, that you could also set up in some 0gc way easily and 
move position and limit on the buffers on every call to `next`).

This interface can easily be used to generate the current 
`org.apache.kafka.clients.consumer.ConsumerRecords` in an efficient way (but 
would also if made public) allow users to implement more efficient 
serialization/deserialization approaches on top. Examples that would massively 
profit from this approach would be Avro or Hadoop's Writables, that could keep 
reusing the same key and value objects without the need to constantly run 
expensive `byte[]` allocations.

Having this as the basis of the current implementation and publicly exposed 
would deal with this concern you voiced:

{code}
You could potentially implement both but you would need to change consumer.poll 
to allow passing back in the ConsumerRecords instance for reuse when you are 
done with it.
{code}

-> no need for this then in my opinion. Users can just come up with their own 
way to handle memory. Passing back records that aren't needed anymore seems 
like less than practical to implement. Much easier if this is just done by 
direct reuse of a single object like in Hadoop, Avro or Spark.

Also 

{code}
2. We don't mangle the code to badly in doing so
{code}

wouldn't be much of a concern either imo since you can keep the old interface 
around on top. When implementing this I'd simply try to get this interface in 
at the lowest possible level and keep the existing codebase on top. This would 
probably still allow some optimization in the existing code on top + it would 
give uses the ability to write much faster consumers.
Again kind of like Avor has it with giving you the option to reuse and object 
when deserializing, but still keeping the slower API that gives you a new one 
one every `next` call around on top of things 
(https://avro.apache.org/docs/current/gettingstartedjava.html#Deserializing is 
what I mean).

PS: If this is seen as a viable approach I'd be happy to give it a go. This 
should be doable for me in a "not overly invasive to the existing codebase" way 
I think.

EDIT:

- removed `close()` and `poll()` from interface, it makes no sense to extract 
the memory handling to outside the consumer, when we have the old api that 
needs to be supported too and requires the same reusable buffering



was (Author: original-brownbear):
[~jkreps] Maybe let me give the perspective of a user, using Kafka in a 
situation where high throughput and a low GC footprint are key and this change 
would be extremely valuable.

What I would ideally like to see here and in KAFKA-2045 is this:

Create a lower level interface along the lines of Hadoop's 
[RawKeyValueIterator](http://atetric.com/atetric/javadoc/org.apache.hadoop/hadoop-mapreduce-client-core/2.6.2/src-html/org/apache/hadoop/mapred/RawKeyValueIterator.html)
 that would back the current KafkaConsumer interface:

{code}public interface RawRecordIterator {
/**
 * Gets the current raw key.
 *
 * @return Gets the current raw key
 * @throws IOException
 */
ByteBuffer getKey() throws IOException;

/**
 * Gets the 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

If the argument for not having a map holding the key, value pairs is due to 
garbage creation of HashMap entry's, forcing the creation of a whole new 
producer record to simply add a head, surely is creating a-lot more?

From: Jason Gustafson 
Sent: Wednesday, February 22, 2017 10:09 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers

The current producer interceptor API is this:

ProducerRecord onSend(ProducerRecord record);

So adding a header means creating a new ProducerRecord with a new header
added to the current headers and returning it. Would that not work?

-Jason

On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce 
wrote:

> So how would you have this work if not mutable where interceptors would
> add headers?
>
> Sent using OWA for iPhone
> 
> From: Jason Gustafson 
> Sent: Wednesday, February 22, 2017 8:42:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> I think the point on the mutability of Headers is worth discussing a little
> more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
> constructed, there should be no need to further change the headers. Is that
> correct? If so, then why not enforce that that is the case through the API?
> One problem with mutability it that it constrains the implementation of
> Headers. For example, if we were backing with a byte slice, would we recopy
> the bytes if a header is added or would we maintain a satellite collection
> of added records. Seems not great either way. If we really think mutability
> is needed, perhaps we could add a method to headers to convert it to a
> mutable type (e.g. a Headers.toMap)?
>
> I'm also with Ismael about exposing Headers.get(). I thought it might make
> sense to have a method like this instead:
>
> Iterable findMatching(Pattern pattern);
>
> This makes the (potential) need to scan the headers clear in the API. I'd
> also be fine exposing no getter at all. In general, Ï think it's good to
> start with a minimalistic API and work from there since it's always easier
> to add methods than remove them.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
> wrote:
>
> >
> > Hi Ismael
> >
> > On point 1,
> >
> > Sure makes sense will update shortly.
> >
> > On point 2,
> >
> > Setter/getter typical to properties/headers api’s traditionally are map
> > styled interfaces and what I believe is most expected styled thus the
> Key,
> > Value setter.
> > Also it would mean rather than an interface, we would be making our
> > internal header impl object we have for the array, exposed. E.g. if we
> had
> > a Map really this would be Map.Entry interface, this is the same reasons
> on
> > the map interface I cannot actually make the underlying Node object
> that’s
> > the implementation for HashMap, so that internals can be changed.
> >
> >
> > On point 3,
> >
> > I think it people do expect it to be performant, thus why originally
> > concern I raised with using an array for to me is an early memory
> > optimisation. I think the user experience of properties/headers is on a
> > get/set model. This is why its important we have encapsulated logic that
> > then allows us to change this to a map, if this becomes and issue, and
> the
> > memory overhead of hashmap is less so.
> >
> >
> >
> >
> > On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" <
> > isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:
> >
> > Hi all,
> >
> > Great to see the progress that has been achieved on this one. :) A
> few
> > comments regarding the APIs (I'm still reviewing the message format
> > changes):
> >
> > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should
> be
> > named `headers` (we avoid the `get` prefix in Kafka)
> >
> > 2. The `Headers` class is mutable (there's an `add` method). Does it
> > need
> > to be? If so, it would be good to explain why. Related to that, we
> > should
> > also explain the thinking around thread-safety. If we keep the `add`
> > method, it may make sense for it to take a `Header` (that way we can
> > add
> > things to `Header` without changing the interface).
> >
> > 3. Do we need the `Headers.get()` method? People usually assume that
> > `get`
> > would be efficient, but depending on the implementation (the current
> > proposal states that an array would be used), it may not be. If we
> > expect
> > the number of headers to be small, it doesn't matter though.
> >
> > Ismael
> >
> > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> michael.pea...@ig.com
> > >
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Have converted the interface/api bullets into interface code
> > snippets.
> > >
> > > Agreed implementation won’t take too 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Gwen Shapira]

I saw them in Maven yesterday?

On Wed, Feb 22, 2017 at 2:15 PM, Stephane Maarek
 wrote:
> Awesome thanks a lot! When should we expect the dependencies to be released
> in Maven? (including 2.12 scala)
>
> On 23 February 2017 at 8:27:10 am, Jun Rao (j...@confluent.io) wrote:
>
> Thanks for driving the release, Ewen.
>
> Jun
>
> On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava 
> wrote:
>
>> The Apache Kafka community is pleased to announce the release for Apache
>> Kafka 0.10.2.0. This is a feature release which includes the completion
>> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
>> requests merged.
>>
>> All of the changes in this release can be found in the release notes:
>> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
>>
>> Apache Kafka is a distributed streaming platform with four four core
>> APIs:
>>
>> ** The Producer API allows an application to publish a stream records to
>> one or more Kafka topics.
>>
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>>
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an
>> output
>> stream to one or more output topics, effectively transforming the input
>> streams to output streams.
>>
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might capture
>> every change to a table.three key capabilities:
>>
>>
>> With these APIs, Kafka can be used for two broad classes of application:
>>
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>>
>> ** Building real-time streaming applications that transform or react to
>> the
>> streams of data.
>>
>>
>> You can download the source release from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka-0.10.2.0-src.tgz
>>
>> and binary releases from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.11-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.10-0.10.2.0.tgz
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
>> 0/kafka_2.12-0.10.2.0.tgz
>> (experimental 2.12 artifact)
>>
>> Thanks to the 101 contributors on this release!
>>
>> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
>> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
>> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
>> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
>> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
>> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
>> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
>> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
>> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
>> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
>> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
>> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
>> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
>> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
>> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
>> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
>> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
>> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
>> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
>> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
>> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
>> Yang Wei, yaojuncn, Yuto Kawamura
>>
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at
>> http://kafka.apache.org/
>>
>> Thanks,
>> Ewen
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Jenkins build is back to normal : kafka-0.10.2-jdk7 #86

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-124: Request rate quotas

[Rajini Sivaram]

Jun/Roger,

Thank you for the feedback.

1. I have updated the KIP to use absolute units instead of percentage. The
property is called* io_thread_units* to align with the thread count
property *num.io.threads*. When we implement network thread utilization
quotas, we can add another property *network_thread_units.*

2. ControlledShutdown is already listed under the exempt requests. Jun, did
you mean a different request that needs to be added? The four requests
currently exempt in the KIP are StopReplica, ControlledShutdown,
LeaderAndIsr and UpdateMetadata. These are controlled using ClusterAction
ACL, so it is easy to exclude and only throttle if unauthorized. I wasn't
sure if there are other requests used only for inter-broker that needed to
be excluded.

3. I was thinking the smallest change would be to replace all references to
*requestChannel.sendResponse()* with a local method
*sendResponseMaybeThrottle()* that does the throttling if any plus send
response. If we throttle first in *KafkaApis.handle()*, the time spent
within the method handling the request will not be recorded or used in
throttling. We can look into this again when the PR is ready for review.

Regards,

Rajini



On Wed, Feb 22, 2017 at 5:55 PM, Roger Hoover 
wrote:

> Great to see this KIP and the excellent discussion.
>
> To me, Jun's suggestion makes sense.  If my application is allocated 1
> request handler unit, then it's as if I have a Kafka broker with a single
> request handler thread dedicated to me.  That's the most I can use, at
> least.  That allocation doesn't change even if an admin later increases the
> size of the request thread pool on the broker.  It's similar to the CPU
> abstraction that VMs and containers get from hypervisors or OS schedulers.
> While different client access patterns can use wildly different amounts of
> request thread resources per request, a given application will generally
> have a stable access pattern and can figure out empirically how many
> "request thread units" it needs to meet it's throughput/latency goals.
>
> Cheers,
>
> Roger
>
> On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao  wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the updated KIP. A few more comments.
> >
> > 1. A concern of request_time_percent is that it's not an absolute value.
> > Let's say you give a user a 10% limit. If the admin doubles the number of
> > request handler threads, that user now actually has twice the absolute
> > capacity. This may confuse people a bit. So, perhaps setting the quota
> > based on an absolute request thread unit is better.
> >
> > 2. ControlledShutdownRequest is also an inter-broker request and needs to
> > be excluded from throttling.
> >
> > 3. Implementation wise, I am wondering if it's simpler to apply the
> request
> > time throttling first in KafkaApis.handle(). Otherwise, we will need to
> add
> > the throttling logic in each type of request.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram  >
> > wrote:
> >
> > > Jun,
> > >
> > > Thank you for the review.
> > >
> > > I have reverted to the original KIP that throttles based on request
> > handler
> > > utilization. At the moment, it uses percentage, but I am happy to
> change
> > to
> > > a fraction (out of 1 instead of 100) if required. I have added the
> > examples
> > > from this discussion to the KIP. Also added a "Future Work" section to
> > > address network thread utilization. The configuration is named
> > > "request_time_percent" with the expectation that it can also be used as
> > the
> > > limit for network thread utilization when that is implemented, so that
> > > users have to set only one config for the two and not have to worry
> about
> > > the internal distribution of the work between the two thread pools in
> > > Kafka.
> > >
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao  wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > Thanks for the proposal.
> > > >
> > > > The benefit of using the request processing time over the request
> rate
> > is
> > > > exactly what people have said. I will just expand that a bit.
> Consider
> > > the
> > > > following case. The producer sends a produce request with a 10MB
> > message
> > > > but compressed to 100KB with gzip. The decompression of the message
> on
> > > the
> > > > broker could take 10-15 seconds, during which time, a request handler
> > > > thread is completely blocked. In this case, neither the byte-in quota
> > nor
> > > > the request rate quota may be effective in protecting the broker.
> > > Consider
> > > > another case. A consumer group starts with 10 instances and later on
> > > > switches to 20 instances. The request rate will likely double, but
> the
> > > > actually load on the broker may not double since each fetch request
> > only
> > > > contains half of the partitions. Request rate quota may 

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Prateek Jaipuria (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879358#comment-15879358
 ] 

Prateek Jaipuria commented on KAFKA-2729:
-

[~granthenke] We don't see any zookeeper disconnections.

Just
{code}
INFO Partition [topic,n] on broker m: Cached zkVersion [xxx] not equal to that 
in zookeeper, skip updating ISR (kafka.cluster.Partition)
{code}

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

The current producer interceptor API is this:

ProducerRecord onSend(ProducerRecord record);

So adding a header means creating a new ProducerRecord with a new header
added to the current headers and returning it. Would that not work?

-Jason

On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce 
wrote:

> So how would you have this work if not mutable where interceptors would
> add headers?
>
> Sent using OWA for iPhone
> 
> From: Jason Gustafson 
> Sent: Wednesday, February 22, 2017 8:42:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> I think the point on the mutability of Headers is worth discussing a little
> more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
> constructed, there should be no need to further change the headers. Is that
> correct? If so, then why not enforce that that is the case through the API?
> One problem with mutability it that it constrains the implementation of
> Headers. For example, if we were backing with a byte slice, would we recopy
> the bytes if a header is added or would we maintain a satellite collection
> of added records. Seems not great either way. If we really think mutability
> is needed, perhaps we could add a method to headers to convert it to a
> mutable type (e.g. a Headers.toMap)?
>
> I'm also with Ismael about exposing Headers.get(). I thought it might make
> sense to have a method like this instead:
>
> Iterable findMatching(Pattern pattern);
>
> This makes the (potential) need to scan the headers clear in the API. I'd
> also be fine exposing no getter at all. In general, Ï think it's good to
> start with a minimalistic API and work from there since it's always easier
> to add methods than remove them.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
> wrote:
>
> >
> > Hi Ismael
> >
> > On point 1,
> >
> > Sure makes sense will update shortly.
> >
> > On point 2,
> >
> > Setter/getter typical to properties/headers api’s traditionally are map
> > styled interfaces and what I believe is most expected styled thus the
> Key,
> > Value setter.
> > Also it would mean rather than an interface, we would be making our
> > internal header impl object we have for the array, exposed. E.g. if we
> had
> > a Map really this would be Map.Entry interface, this is the same reasons
> on
> > the map interface I cannot actually make the underlying Node object
> that’s
> > the implementation for HashMap, so that internals can be changed.
> >
> >
> > On point 3,
> >
> > I think it people do expect it to be performant, thus why originally
> > concern I raised with using an array for to me is an early memory
> > optimisation. I think the user experience of properties/headers is on a
> > get/set model. This is why its important we have encapsulated logic that
> > then allows us to change this to a map, if this becomes and issue, and
> the
> > memory overhead of hashmap is less so.
> >
> >
> >
> >
> > On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" <
> > isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:
> >
> > Hi all,
> >
> > Great to see the progress that has been achieved on this one. :) A
> few
> > comments regarding the APIs (I'm still reviewing the message format
> > changes):
> >
> > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should
> be
> > named `headers` (we avoid the `get` prefix in Kafka)
> >
> > 2. The `Headers` class is mutable (there's an `add` method). Does it
> > need
> > to be? If so, it would be good to explain why. Related to that, we
> > should
> > also explain the thinking around thread-safety. If we keep the `add`
> > method, it may make sense for it to take a `Header` (that way we can
> > add
> > things to `Header` without changing the interface).
> >
> > 3. Do we need the `Headers.get()` method? People usually assume that
> > `get`
> > would be efficient, but depending on the implementation (the current
> > proposal states that an array would be used), it may not be. If we
> > expect
> > the number of headers to be small, it doesn't matter though.
> >
> > Ismael
> >
> > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> michael.pea...@ig.com
> > >
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Have converted the interface/api bullets into interface code
> > snippets.
> > >
> > > Agreed implementation won’t take too long. We have early versions
> > already.
> > > Maybe a week before you think about merging I would assume it would
> > be more
> > > stabilised? I was thinking then we could fork from your confluent
> > branch,
> > > making and then holding KIP-82 changes in a patch file, that we can
> > then
> > > re-fork from apache once KIP98 final is merged, and apply patch
> with
> > last
> > > minute changes.
> >

[jira] [Commented] (KAFKA-4787) KafkaStreams close() is not reentrant

[Steven Schlansker (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879336#comment-15879336
 ] 

Steven Schlansker commented on KAFKA-4787:
--

I'm not sure I understand why that should behave differently.  Replacing the 
lambda expression ({{this::handleStreamException}}) with your anonymous 
subclass implementation should be equivalent with regards to taking the 
{{synchronized}} monitor of the KafkaStreams object inside the {{public 
synchronized void close(long, TimeUnit)}} method.

I will test it anyway though and see if it in fact does behave differently.

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Mayuresh Gharat]

Hi Jun,

Thanks a lot for the comments and reviews.
I agree we should log the username.
What I meant by creating KafkaPrincipal was, after this KIP we would not be
required to create KafkaPrincipal and if we want to maintain the old
logging, we will have to create it as we do today.
I will take care that we specify the Principal name in the log.

Thanks again for all the reviews.

Thanks,

Mayuresh

On Wed, Feb 22, 2017 at 1:45 PM, Jun Rao  wrote:

> Hi, Mayuresh,
>
> For logging the user name, we could do either way. We just need to make
> sure the expected user name is logged. Also, currently, we are already
> creating a KafkaPrincipal on every request. +1 on the latest KIP.
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Jun,
> >
> > Thanks for the comments.
> >
> > I will mention in the KIP : how this change doesn't affect the default
> > authorizer implementation.
> >
> > Regarding, Currently, we log the principal name in the request log in
> > RequestChannel, which has the format of "principalType + SEPARATOR +
> > name;".
> > It would be good if we can keep the same convention after this KIP. One
> way
> > to do that is to convert java.security.Principal to KafkaPrincipal for
> > logging the requests.
> > --- > This would mean we have to create a new KafkaPrincipal on each
> > request. Would it be OK to just specify the name of the principal.
> > Is there any major reason, we don't want to change the logging format?
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> >
> > On Mon, Feb 20, 2017 at 10:18 PM, Jun Rao  wrote:
> >
> > > Hi, Mayuresh,
> > >
> > > Thanks for the updated KIP. A couple of more comments.
> > >
> > > 1. Do we convert java.security.Principal to KafkaPrincipal for
> > > authorization check in SimpleAclAuthorizer? If so, it would be useful
> to
> > > mention that in the wiki so that people can understand how this change
> > > doesn't affect the default authorizer implementation.
> > >
> > > 2. Currently, we log the principal name in the request log in
> > > RequestChannel, which has the format of "principalType + SEPARATOR +
> > > name;".
> > > It would be good if we can keep the same convention after this KIP. One
> > way
> > > to do that is to convert java.security.Principal to KafkaPrincipal for
> > > logging the requests.
> > >
> > > Jun
> > >
> > >
> > > On Fri, Feb 17, 2017 at 5:35 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com
> > > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > I have updated the KIP. Would you mind taking another look?
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Fri, Feb 17, 2017 at 4:42 PM, Mayuresh Gharat <
> > > > gharatmayures...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Sure sounds good to me.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > > On Fri, Feb 17, 2017 at 1:54 PM, Jun Rao  wrote:
> > > > >
> > > > >> Hi, Mani,
> > > > >>
> > > > >> Good point on using PrincipalBuilder for SASL. It seems that
> > > > >> PrincipalBuilder already has access to Authenticator. So, we could
> > > just
> > > > >> enable that in SaslChannelBuilder. We probably could do that in a
> > > > separate
> > > > >> KIP?
> > > > >>
> > > > >> Hi, Mayuresh,
> > > > >>
> > > > >> If you don't think there is a concrete use case for using
> > > > >> PrincipalBuilder in
> > > > >> kafka-acls.sh, perhaps we could do the simpler approach for now?
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Feb 17, 2017 at 12:23 PM, Mayuresh Gharat <
> > > > >> gharatmayures...@gmail.com> wrote:
> > > > >>
> > > > >> > @Manikumar,
> > > > >> >
> > > > >> > Can you give an example how you are planning to use
> > > PrincipalBuilder?
> > > > >> >
> > > > >> > @Jun
> > > > >> > Yes, that is right. To give a brief overview, we just extract
> the
> > > cert
> > > > >> and
> > > > >> > hand it over to a third party library for creating a Principal.
> So
> > > we
> > > > >> > cannot create a Principal from just a string.
> > > > >> > The main motive behind adding the PrincipalBuilder for
> > kafk-acls.sh
> > > > was
> > > > >> > that someone else (who can generate a Principal from map of
> > > propertie,
> > > > >> >  for example) can use it.
> > > > >> > As I said, Linkedin is fine with not making any changes to
> > > > Kafka-acls.sh
> > > > >> > for now. But we thought that it would be a good improvement to
> the
> > > > tool
> > > > >> and
> > > > >> > it makes it more flexible and usable.
> > > > >> >
> > > > >> > Let us know your thoughts, if you would like us to make
> > > kafka-acls.sh
> > > > >> more
> > > > >> > flexible and usable and not limited to Authorizer coming out of
> > the
> > > > box.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Mayuresh
> > > > >> >
> > > > >> >
> > > > >> > On 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

So how would you have this work if not mutable where interceptors would add 
headers?

Sent using OWA for iPhone

From: Jason Gustafson 
Sent: Wednesday, February 22, 2017 8:42:27 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers

I think the point on the mutability of Headers is worth discussing a little
more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
constructed, there should be no need to further change the headers. Is that
correct? If so, then why not enforce that that is the case through the API?
One problem with mutability it that it constrains the implementation of
Headers. For example, if we were backing with a byte slice, would we recopy
the bytes if a header is added or would we maintain a satellite collection
of added records. Seems not great either way. If we really think mutability
is needed, perhaps we could add a method to headers to convert it to a
mutable type (e.g. a Headers.toMap)?

I'm also with Ismael about exposing Headers.get(). I thought it might make
sense to have a method like this instead:

Iterable findMatching(Pattern pattern);

This makes the (potential) need to scan the headers clear in the API. I'd
also be fine exposing no getter at all. In general, Ï think it's good to
start with a minimalistic API and work from there since it's always easier
to add methods than remove them.

-Jason

On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
wrote:

>
> Hi Ismael
>
> On point 1,
>
> Sure makes sense will update shortly.
>
> On point 2,
>
> Setter/getter typical to properties/headers api’s traditionally are map
> styled interfaces and what I believe is most expected styled thus the Key,
> Value setter.
> Also it would mean rather than an interface, we would be making our
> internal header impl object we have for the array, exposed. E.g. if we had
> a Map really this would be Map.Entry interface, this is the same reasons on
> the map interface I cannot actually make the underlying Node object that’s
> the implementation for HashMap, so that internals can be changed.
>
>
> On point 3,
>
> I think it people do expect it to be performant, thus why originally
> concern I raised with using an array for to me is an early memory
> optimisation. I think the user experience of properties/headers is on a
> get/set model. This is why its important we have encapsulated logic that
> then allows us to change this to a map, if this becomes and issue, and the
> memory overhead of hashmap is less so.
>
>
>
>
> On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" <
> isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:
>
> Hi all,
>
> Great to see the progress that has been achieved on this one. :) A few
> comments regarding the APIs (I'm still reviewing the message format
> changes):
>
> 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
> named `headers` (we avoid the `get` prefix in Kafka)
>
> 2. The `Headers` class is mutable (there's an `add` method). Does it
> need
> to be? If so, it would be good to explain why. Related to that, we
> should
> also explain the thinking around thread-safety. If we keep the `add`
> method, it may make sense for it to take a `Header` (that way we can
> add
> things to `Header` without changing the interface).
>
> 3. Do we need the `Headers.get()` method? People usually assume that
> `get`
> would be efficient, but depending on the implementation (the current
> proposal states that an array would be used), it may not be. If we
> expect
> the number of headers to be small, it doesn't matter though.
>
> Ismael
>
> On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce  >
> wrote:
>
> > Hi Jason,
> >
> > Have converted the interface/api bullets into interface code
> snippets.
> >
> > Agreed implementation won’t take too long. We have early versions
> already.
> > Maybe a week before you think about merging I would assume it would
> be more
> > stabilised? I was thinking then we could fork from your confluent
> branch,
> > making and then holding KIP-82 changes in a patch file, that we can
> then
> > re-fork from apache once KIP98 final is merged, and apply patch with
> last
> > minute changes.
> >
> > Cheers
> > Mike
> >
> >
> > On 22/02/2017, 00:56, "Jason Gustafson"  wrote:
> >
> > Hey Michael,
> >
> > Awesome. I have a minor request. The APIs are currently
> documented as a
> > wiki list. Would you mind adding a code snippet instead? It's a
> bit
> > easier
> > to process.
> >
> > How will be best to manage this, as we will obviously build off
> your
> > KIP’s
> > > protocol changes, to avoid a merge hell, should we branch from
> your
> > branch
> 

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Jun Rao]

Hi, Mayuresh,

For logging the user name, we could do either way. We just need to make
sure the expected user name is logged. Also, currently, we are already
creating a KafkaPrincipal on every request. +1 on the latest KIP.

Thanks,

Jun


On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat  wrote:

> Hi Jun,
>
> Thanks for the comments.
>
> I will mention in the KIP : how this change doesn't affect the default
> authorizer implementation.
>
> Regarding, Currently, we log the principal name in the request log in
> RequestChannel, which has the format of "principalType + SEPARATOR +
> name;".
> It would be good if we can keep the same convention after this KIP. One way
> to do that is to convert java.security.Principal to KafkaPrincipal for
> logging the requests.
> --- > This would mean we have to create a new KafkaPrincipal on each
> request. Would it be OK to just specify the name of the principal.
> Is there any major reason, we don't want to change the logging format?
>
> Thanks,
>
> Mayuresh
>
>
>
> On Mon, Feb 20, 2017 at 10:18 PM, Jun Rao  wrote:
>
> > Hi, Mayuresh,
> >
> > Thanks for the updated KIP. A couple of more comments.
> >
> > 1. Do we convert java.security.Principal to KafkaPrincipal for
> > authorization check in SimpleAclAuthorizer? If so, it would be useful to
> > mention that in the wiki so that people can understand how this change
> > doesn't affect the default authorizer implementation.
> >
> > 2. Currently, we log the principal name in the request log in
> > RequestChannel, which has the format of "principalType + SEPARATOR +
> > name;".
> > It would be good if we can keep the same convention after this KIP. One
> way
> > to do that is to convert java.security.Principal to KafkaPrincipal for
> > logging the requests.
> >
> > Jun
> >
> >
> > On Fri, Feb 17, 2017 at 5:35 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi Jun,
> > >
> > > I have updated the KIP. Would you mind taking another look?
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Fri, Feb 17, 2017 at 4:42 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com
> > > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Sure sounds good to me.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Fri, Feb 17, 2017 at 1:54 PM, Jun Rao  wrote:
> > > >
> > > >> Hi, Mani,
> > > >>
> > > >> Good point on using PrincipalBuilder for SASL. It seems that
> > > >> PrincipalBuilder already has access to Authenticator. So, we could
> > just
> > > >> enable that in SaslChannelBuilder. We probably could do that in a
> > > separate
> > > >> KIP?
> > > >>
> > > >> Hi, Mayuresh,
> > > >>
> > > >> If you don't think there is a concrete use case for using
> > > >> PrincipalBuilder in
> > > >> kafka-acls.sh, perhaps we could do the simpler approach for now?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Feb 17, 2017 at 12:23 PM, Mayuresh Gharat <
> > > >> gharatmayures...@gmail.com> wrote:
> > > >>
> > > >> > @Manikumar,
> > > >> >
> > > >> > Can you give an example how you are planning to use
> > PrincipalBuilder?
> > > >> >
> > > >> > @Jun
> > > >> > Yes, that is right. To give a brief overview, we just extract the
> > cert
> > > >> and
> > > >> > hand it over to a third party library for creating a Principal. So
> > we
> > > >> > cannot create a Principal from just a string.
> > > >> > The main motive behind adding the PrincipalBuilder for
> kafk-acls.sh
> > > was
> > > >> > that someone else (who can generate a Principal from map of
> > propertie,
> > > >> >  for example) can use it.
> > > >> > As I said, Linkedin is fine with not making any changes to
> > > Kafka-acls.sh
> > > >> > for now. But we thought that it would be a good improvement to the
> > > tool
> > > >> and
> > > >> > it makes it more flexible and usable.
> > > >> >
> > > >> > Let us know your thoughts, if you would like us to make
> > kafka-acls.sh
> > > >> more
> > > >> > flexible and usable and not limited to Authorizer coming out of
> the
> > > box.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Mayuresh
> > > >> >
> > > >> >
> > > >> > On Thu, Feb 16, 2017 at 10:18 PM, Manikumar <
> > > manikumar.re...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Jun,
> > > >> > >
> > > >> > > yes, we can just customize rules to send full principal name.  I
> > was
> > > >> > > just thinking to
> > > >> > > use PrinciplaBuilder interface for implementing SASL rules also.
> > So
> > > >> that
> > > >> > > the interface
> > > >> > > will be consistent across protocols.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Fri, Feb 17, 2017 at 1:07 AM, Jun Rao 
> > wrote:
> > > >> > >
> > > >> > > > Hi, Radai, Mayuresh,
> > > >> > > >
> > > >> > > > Thanks for the explanation. Good point on a pluggable
> authorizer
> > > can
> > > >> > > > customize how acls are added. However, 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Jun Rao]

Thanks for driving the release, Ewen.

Jun

On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava 
wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.2.0. This is a feature release which includes the completion
> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> requests merged.
>
> All of the changes in this release can be found in the release notes:
> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
>
> Apache Kafka is a distributed streaming platform with four four core
> APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react to
> the
> streams of data.
>
>
> You can download the source release from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka-0.10.2.0-src.tgz
>
> and binary releases from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.11-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.10-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.12-0.10.2.0.tgz
> (experimental 2.12 artifact)
>
> Thanks to the 101 contributors on this release!
>
> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> Yang Wei, yaojuncn, Yuto Kawamura
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
> Thanks,
> Ewen
>

[jira] [Commented] (KAFKA-4787) KafkaStreams close() is not reentrant

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879270#comment-15879270
 ] 

Matthias J. Sax commented on KAFKA-4787:


You should set the handler like this:

{noformat}
streams.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
public uncaughtException(Thread t, throwable e) {
// here you should examine the exception and perform an appropriate 
action!
}
);
{noformat}

Does this help?

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

[Dave Thomas (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879220#comment-15879220
 ] 

Dave Thomas commented on KAFKA-4778:


[~guozhang] We set the following topic configuration values:
{noformat}
kafka.default.replication.factor=3
kafka.default.partition.count=32
cleanup.policy=compact
retention.bytes=-1
retention.ms=259200 // 30 days for internal streams; 720 days for ingested 
raw streams
{noformat}

Is that what you're looking for?

> OOM on kafka-streams instances with high numbers of unreaped Record classes
> ---
>
> Key: KAFKA-4778
> URL: https://issues.apache.org/jira/browse/KAFKA-4778
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.1.1
> Environment: AWS m3.large Ubuntu 16.04.1 LTS.  rocksDB on local SSD.  
> Kafka has 3 zk, 5 brokers.  
> stream processors are run with:
> -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
> Stream processors written in scala 2.11.8
>Reporter: Dave Thomas
> Attachments: oom-killer.txt
>
>
> We have a stream processing app with ~8 source/sink stages operating roughly 
> at the rate of 500k messages ingested/day (~4M across the 8 stages).
> We get OOM eruptions once every ~18 hours. Note it is Linux triggering the 
> OOM-killer, not the JVM terminating itself. 
> It may be worth noting that stream processing uses ~50 mbytes while 
> processing normally for hours on end, until the problem surfaces; then in 
> ~20-30 sec memory grows suddenly from under 100 mbytes to >1 gig and does not 
> shrink until the process is killed.
> We are using supervisor to restart the instances.  Sometimes, the process 
> dies immediately once stream processing resumes for the same reason, a 
> process which could continue for minutes or hours.  This extended window has 
> enabled us to capture a heap dump using jmap.
> jhat's histogram feature reveals the following top objects in memory:
> Class Instance Count  Total Size
> class [B  4070487 867857833
> class [Ljava.lang.Object; 2066986 268036184
> class [C  539519  92010932
> class [S  1003290 80263028
> class [I  508208  77821516
> class java.nio.HeapByteBuffer 1506943 58770777
> class org.apache.kafka.common.record.Record   1506783 36162792
> class org.apache.kafka.clients.consumer.ConsumerRecord528652  35948336
> class org.apache.kafka.common.record.MemoryRecords$RecordsIterator501742  
> 32613230
> class org.apache.kafka.common.record.LogEntry 2009373 32149968
> class org.xerial.snappy.SnappyInputStream 501600  20565600
> class java.io.DataInputStream 501742  20069680
> class java.io.EOFException501606  20064240
> class java.util.ArrayDeque501941  8031056
> class java.lang.Long  516463  4131704
> Note high on the list include org.apache.kafka.common.record.Record, 
> org.apache.kafka.clients.consumer.ConsumerRecord,
> org.apache.kafka.common.record.MemoryRecords$RecordsIterator,
> org.apache.kafka.common.record.LogEntry
> All of these contain 500k-1.5M instances.
> There is nothing in stream processing logs that is distinctive (log levels 
> are still at default).
> Could it be references (weak, phantom, etc) causing these instances to not be 
> garbage-collected?
> Edit: to request a full heap dump (created using `jmap 
> -dump:format=b,file=`), contact me directly at opensou...@peoplemerge.com.  
> It is 2G.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Dave Thomas (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879213#comment-15879213
 ] 

Dave Thomas commented on KAFKA-2729:


[~granthenke] We don't see brokers recovering.  The message we see is:
{noformat}
Cached zkVersion [xxx] not equal to that in zookeeper, skip updating ISR
{noformat}



> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-4786) Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4786.

   Resolution: Fixed
Fix Version/s: 0.10.2.1
   0.10.3.0

Issue resolved by pull request 2586
[https://github.com/apache/kafka/pull/2586]

> Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose
> 
>
> Key: KAFKA-4786
> URL: https://issues.apache.org/jira/browse/KAFKA-4786
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Rajini Sivaram
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> I've seen this failure in Jenkins a number of times:
> {code}
> java.lang.AssertionError: Heartbeat thread active after close
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertFalse(Assert.java:64)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testHeartbeatThreadClose(ConsumerCoordinatorTest.java:1425)
> {code}
> https://jenkins.confluent.io/job/kafka-0.10.2/88/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testHeartbeatThreadClose/
> [~rsivaram], do you think you can take a look since you wrote the test?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jason Gustafson]

I think the point on the mutability of Headers is worth discussing a little
more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
constructed, there should be no need to further change the headers. Is that
correct? If so, then why not enforce that that is the case through the API?
One problem with mutability it that it constrains the implementation of
Headers. For example, if we were backing with a byte slice, would we recopy
the bytes if a header is added or would we maintain a satellite collection
of added records. Seems not great either way. If we really think mutability
is needed, perhaps we could add a method to headers to convert it to a
mutable type (e.g. a Headers.toMap)?

I'm also with Ismael about exposing Headers.get(). I thought it might make
sense to have a method like this instead:

Iterable findMatching(Pattern pattern);

This makes the (potential) need to scan the headers clear in the API. I'd
also be fine exposing no getter at all. In general, Ï think it's good to
start with a minimalistic API and work from there since it's always easier
to add methods than remove them.

-Jason

On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce 
wrote:

>
> Hi Ismael
>
> On point 1,
>
> Sure makes sense will update shortly.
>
> On point 2,
>
> Setter/getter typical to properties/headers api’s traditionally are map
> styled interfaces and what I believe is most expected styled thus the Key,
> Value setter.
> Also it would mean rather than an interface, we would be making our
> internal header impl object we have for the array, exposed. E.g. if we had
> a Map really this would be Map.Entry interface, this is the same reasons on
> the map interface I cannot actually make the underlying Node object that’s
> the implementation for HashMap, so that internals can be changed.
>
>
> On point 3,
>
> I think it people do expect it to be performant, thus why originally
> concern I raised with using an array for to me is an early memory
> optimisation. I think the user experience of properties/headers is on a
> get/set model. This is why its important we have encapsulated logic that
> then allows us to change this to a map, if this becomes and issue, and the
> memory overhead of hashmap is less so.
>
>
>
>
> On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" <
> isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:
>
> Hi all,
>
> Great to see the progress that has been achieved on this one. :) A few
> comments regarding the APIs (I'm still reviewing the message format
> changes):
>
> 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
> named `headers` (we avoid the `get` prefix in Kafka)
>
> 2. The `Headers` class is mutable (there's an `add` method). Does it
> need
> to be? If so, it would be good to explain why. Related to that, we
> should
> also explain the thinking around thread-safety. If we keep the `add`
> method, it may make sense for it to take a `Header` (that way we can
> add
> things to `Header` without changing the interface).
>
> 3. Do we need the `Headers.get()` method? People usually assume that
> `get`
> would be efficient, but depending on the implementation (the current
> proposal states that an array would be used), it may not be. If we
> expect
> the number of headers to be small, it doesn't matter though.
>
> Ismael
>
> On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce  >
> wrote:
>
> > Hi Jason,
> >
> > Have converted the interface/api bullets into interface code
> snippets.
> >
> > Agreed implementation won’t take too long. We have early versions
> already.
> > Maybe a week before you think about merging I would assume it would
> be more
> > stabilised? I was thinking then we could fork from your confluent
> branch,
> > making and then holding KIP-82 changes in a patch file, that we can
> then
> > re-fork from apache once KIP98 final is merged, and apply patch with
> last
> > minute changes.
> >
> > Cheers
> > Mike
> >
> >
> > On 22/02/2017, 00:56, "Jason Gustafson"  wrote:
> >
> > Hey Michael,
> >
> > Awesome. I have a minor request. The APIs are currently
> documented as a
> > wiki list. Would you mind adding a code snippet instead? It's a
> bit
> > easier
> > to process.
> >
> > How will be best to manage this, as we will obviously build off
> your
> > KIP’s
> > > protocol changes, to avoid a merge hell, should we branch from
> your
> > branch
> > > in the confluent repo or is it worth having a KIP-98 special
> branch
> > in the
> > > apache git, that we can branch/fork from?
> >
> >
> > I was thinking about this also. Ideally we'd like to get the
> changes
> > in as
> > close together as possible since we only want 

[jira] [Commented] (KAFKA-4786) Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879173#comment-15879173
 ] 

ASF GitHub Bot commented on KAFKA-4786:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2586


> Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose
> 
>
> Key: KAFKA-4786
> URL: https://issues.apache.org/jira/browse/KAFKA-4786
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>
> I've seen this failure in Jenkins a number of times:
> {code}
> java.lang.AssertionError: Heartbeat thread active after close
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertFalse(Assert.java:64)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testHeartbeatThreadClose(ConsumerCoordinatorTest.java:1425)
> {code}
> https://jenkins.confluent.io/job/kafka-0.10.2/88/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testHeartbeatThreadClose/
> [~rsivaram], do you think you can take a look since you wrote the test?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879170#comment-15879170
 ] 

Armin Braun commented on KAFKA-1895:


[~hachikuji]

{quote}

How would you propose to expose RawRecordIterator in the consumer API? The 
problem is that IO is currently driven through the poll() API, so introducing 
something else would be a bit weird (though maybe there's a nice way to do it).

{quote}

I admit this is potentially not ideal in terms of nice API design but I can see 
a relatively short route to implementation when adding a method like below to 
`org.apache.kafka.clients.consumer.Consumer`:

{code}
public RawRecordIterator pollRaw(long timeout);
{code}
 
which would simply return the RawRecordIterator that you could also use to back 
the current:

{code}
/**
 * @see KafkaConsumer#poll(long)
 */
public ConsumerRecords poll(long timeout);
{code}

... then you could deserialize the records from the `RawRecordIterator`  when 
instantiating `ConsumerRecords` from the `RawRecordIterator`.
I would also just deserialize when instantiating `ConsumerRecords` from 
`RawRecordIterator` and have that drain the full `RawRecordIterator` to support:

{code}
/**
 * Get just the records for the given partition
 * 
 * @param partition The partition to get records for
 */
public List> records(TopicPartition partition) {
List> recs = this.records.get(partition);
if (recs == null)
return Collections.emptyList();
else
return Collections.unmodifiableList(recs);
}
{code}

still. So no API change needed here + I think moving to a reused set of 
`ByteBuffer` abstracted by `RawRecordIterator` instead of the current solution 
constantly allocating `ByteBuffer` wrapped in things like 
`org.apache.kafka.common.network.NetworkReceive` would be significantly faster 
by saving lots of system calls (wouldn't have to read the header(size) bytes 
separately anymore if pointers on the underlying buffers are used instead of 
setting up buffers and hence byte[] after making that extra system call just to 
read 4 bytes).

So in a nutshell, just allow `pollRaw` which would always return a 
`RawRecordIterator`. Retain all existing APIs and instantiate `ConsumerRecords` 
from the iterator in a (copying) way that keeps `ConsumerRecord` immutable.

Hope this was understandable :) I understand there are a few tricky spots here 
in the implementation (though given that a wealth of work on this exists 
especially in Spark it's not that hard in 2017 imo), but just from the API I 
think the approach is sound. Plus it doesn't require any trickery where 
`ConsumerRecord` instances either become mutable or must be "passed back 
somewhere" to make them reusable, while still offering a 0 GC raw API.


> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2586: KAFKA-4786: Wait for heartbeat thread to terminate...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879124#comment-15879124
 ] 

Grant Henke commented on KAFKA-2729:


I am curious if everyone on this Jira is actually seeing the reported issue. I 
have had multiple cases where someone presented my with an environment they 
thought was experiencing this issue. After researching the environment and 
logs, to date it has always been something else. 

The main culprits so far have been:
* Long GC pauses causing zookeeper sessions to timeout
* Slow or poorly configured zookeeper
* Bad network configuration

All of the above resulted in a soft reoccurring failure of brokers. That churn 
often caused addition load perpetuating the issue. 

If you are seeing this issue do you see the following pattern repeating in the 
logs?:
{noformat}
INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (Disconnected)
...
INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (Expired)
INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, 
session 0x153ab38abdbd360 has expired, closing socket connection
...
INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
INFO kafka.server.KafkaHealthcheck: re-registering broker info in ZK for broker 
32
INFO kafka.utils.ZKCheckedEphemeral: Creating /brokers/ids/32 (is it secure? 
false)
INFO kafka.utils.ZKCheckedEphemeral: Result of znode creation is: OK
{noformat}

If so, something is causing communication with zookeeper to take too long and 
the broker is unregistering itself. This will cause ISRs to shrink and expand 
over and over again.

I don't think this will solve everyones issue here, but hopefully it will help 
solve some.



> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[Prateek Jaipuria (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879076#comment-15879076
 ] 

Prateek Jaipuria commented on KAFKA-2729:
-

Having the same issue with 0.10.1.0 on a 8 node cluster. Restarting node also 
does not help, the problem just moves on to another node. This is becoming a 
deal breaker. Definitely losing trust in Kafka. Definitely a BLOCKER!

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3798) Kafka Consumer 0.10.0.0 killed after rebalancing exception

[Shannon Carey (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879059#comment-15879059
 ] 

Shannon Carey commented on KAFKA-3798:
--

Recently had this problem with an 0.8.2.2 consumer as well. I didn't see any 
Zookeeper connection timeout though. It looks like this was precipitated by 
several Kafka servers restarting at the same time. I don't know why the topic 
was deleted from ZK at one point. Here's an overview of the log messages:

[...-1486594733560-c90a4419], begin rebalancing consumer 
...-1486594733560-c90a4419 try #0"
[...-1486594733560-c90a4419], exception during rebalance "
org.I0Itec.zkclient.exception.ZkNoNodeException: 
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /consumers/.../ids/...-1486594733560-c90a4419
end rebalancing consumer ...-1486594733560-c90a4419 try #0
Rebalancing attempt failed. Clearing the cache before the next rebalancing 
operation is triggered
... various stopping/shutting down of threads ...
Stopping all fetchers
Topic for path /brokers/topics/mytopic gets deleted, which should not happen at 
this time
All connections stopped
Cleared all relevant queues for this fetcher
Cleared the data chunks in all the consumer message iterators
Committing all offsets after clearing the fetcher queues
begin rebalancing consumer ...-1486594733560-c90a4419 try #1
...
can't rebalance after 15 retries

Repeats that several times, for ~6hrs before finally gives up for some reason.


> Kafka Consumer 0.10.0.0 killed after rebalancing exception
> --
>
> Key: KAFKA-3798
> URL: https://issues.apache.org/jira/browse/KAFKA-3798
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 0.10.0.0
> Environment: Production
>Reporter: Sahitya Agrawal
>Assignee: Neha Narkhede
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Hi , 
> I have a topic with 100 partitions and 25 consumers. Consumers were working 
> fine up to some time. 
> After some time I see kafka rebalancing exception in the logs. CPU usage is 
> also 100 % at that time. Consumer process got killed after that. 
> Kafka version : 0.10.0.0
> Some Error print from the logs are following:
> kafka.common.ConsumerRebalanceFailedException: prod_ip- can't rebalance 
> after 10 retries
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:670)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$2.run(ZookeeperConsumerConnector.scala:589)
> exception during rebalance
> org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /consumers/prod/ids/prod_ip-***
> at 
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> at 
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1000)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1099)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:1094)
> at kafka.utils.ZkUtils.readData(ZkUtils.scala:542)
> at kafka.consumer.TopicCount$.constructTopicCount(TopicCount.scala:61)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:674)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:646)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:637)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:637)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:637)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:636)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKSessionExpireListener.handleNewSession(ZookeeperConsumerConnector.scala:522)
> at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:735)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: 
> KeeperErrorCode = NoNode for /consumers/prod/ids/prod_ip-**
> at 
> 

Current offset for partition out of range; reset offset

[Ghosh, Achintya (Contractor)]

Hi All,

One of the partitions showing the huge lag(21K) and I see the below error in 
kafkaserver.out  log of one of the kafka nodes.


Current offset 43294 for partition [PROD_TASK_TOPIC_120,10] out of range; reset 
offset to 43293 (kafka.server.ReplicaFetcherThread)


What is the quick solution, its happening in Production? Will that 21K messages 
again be processed which is wrong? Can we restart the kafka nodes or 
applications to resolve the issue?

Thanks
Achintya

[jira] [Comment Edited] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879020#comment-15879020
 ] 

Jason Gustafson edited comment on KAFKA-1895 at 2/22/17 7:27 PM:
-

[~original-brownbear] How would you propose to expose RawRecordIterator in the 
consumer API? The problem is that IO is currently driven through the poll() 
API, so introducing something else would be a bit weird (though maybe there's a 
nice way to do it). I think that's why Jay was suggesting to push 
deserialization into the ConsumerRecords class. It wouldn't require an API 
change. 

That said, there are definitely use cases where lower level access has been 
desired. MM is one of them. It sucks that we need to decompress and recompress. 
To solve that problem, we would need access to the raw data (not just the key 
value). However, I'm not sure that KafkaProducer and KafkaConsumer are the 
right place for this, nor am I sure that it's ultimately something we want to 
support as a user API.




was (Author: hachikuji):
[~original-brownbear] How would you propose to expose RawRecordIterator in the 
consumer API? The problem is that IO is currently driven through the poll() 
API, so introducing something else would be a bit weird (though maybe there's a 
nice way to do it). I think that's why Jay was suggesting to push 
deserialization into the ConsumerRecords class. It wouldn't require an API 
change. 

That said, there are definitely use cases where lower level access has been 
desired. MM is one of them. It sucks that we need to decompress and recompress. 
To solve that problem, we would need access to the raw data (not just the key 
value). However, I'm not sure that KafkaProducer and KafkaConsumer are the 
right place for this, nor am I sure that it's ultimately something we want 
ultimately want to support as a user API.



> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4787) KafkaStreams close() is not reentrant

[Steven Schlansker (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879019#comment-15879019
 ] 

Steven Schlansker commented on KAFKA-4787:
--

A simple solution could simply have further close() calls not call the inner 
synchronized close overload but instead join on the already existing close 
request

> KafkaStreams close() is not reentrant
> -
>
> Key: KAFKA-4787
> URL: https://issues.apache.org/jira/browse/KAFKA-4787
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Steven Schlansker
>
> While building a simple application, I tried to implement a failure policy 
> where any uncaught exception terminates the application until an 
> administrator can evaluate and intervene:
> {code}
> /** Handle any uncaught exception by shutting down the program. */
> private void handleStreamException(Thread thread, Throwable t) {
> LOG.error("stream exception in thread {}", thread, t);
> streams.close();
> }
> streams.setUncaughtExceptionHandler(this::handleStreamException);
> streams.start();
> {code}
> Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
> prone to what looks like a deadlock:
> {code}
> "StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 
> waiting for monitor entry [0x7f54f03ee000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
> - waiting to lock <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> at 
> com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
>  Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)
> "main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
> [0x7f5610f04000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> - locked <0xfd302bf0> (a java.lang.Thread)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
> - locked <0xf171cda8> (a 
> org.apache.kafka.streams.KafkaStreams)
> at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
> at 
> com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
> Source)
> at 
> com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
> at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
> {code}
> Note how the main thread calls close(), which encounters an exception.  It 
> uses a StreamThread to dispatch to the handler, which calls close().  Once it 
> tries to take the monitor, we are left in a position where main is joined on 
> StreamThread-1, but StreamThread-1 is waiting for main to release that 
> monitor.
> Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
> intentional) -- but to make Kafka Streams robust it should handle any 
> sequence of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879020#comment-15879020
 ] 

Jason Gustafson commented on KAFKA-1895:


[~original-brownbear] How would you propose to expose RawRecordIterator in the 
consumer API? The problem is that IO is currently driven through the poll() 
API, so introducing something else would be a bit weird (though maybe there's a 
nice way to do it). I think that's why Jay was suggesting to push 
deserialization into the ConsumerRecords class. It wouldn't require an API 
change. 

That said, there are definitely use cases where lower level access has been 
desired. MM is one of them. It sucks that we need to decompress and recompress. 
To solve that problem, we would need access to the raw data (not just the key 
value). However, I'm not sure that KafkaProducer and KafkaConsumer are the 
right place for this, nor am I sure that it's ultimately something we want 
ultimately want to support as a user API.



> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4787) KafkaStreams close() is not reentrant

[Steven Schlansker (JIRA)]

Steven Schlansker created KAFKA-4787:


 Summary: KafkaStreams close() is not reentrant
 Key: KAFKA-4787
 URL: https://issues.apache.org/jira/browse/KAFKA-4787
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 0.10.2.0
Reporter: Steven Schlansker


While building a simple application, I tried to implement a failure policy 
where any uncaught exception terminates the application until an administrator 
can evaluate and intervene:

{code}
/** Handle any uncaught exception by shutting down the program. */
private void handleStreamException(Thread thread, Throwable t) {
LOG.error("stream exception in thread {}", thread, t);
streams.close();
}

streams.setUncaughtExceptionHandler(this::handleStreamException);
streams.start();
{code}

Unfortunately, because the KafkaStreams#close() method takes a lock, this is 
prone to what looks like a deadlock:

{code}
"StreamThread-1" #80 prio=5 os_prio=0 tid=0x7f56096f4000 nid=0x40c8 waiting 
for monitor entry [0x7f54f03ee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java)
- waiting to lock <0xf171cda8> (a 
org.apache.kafka.streams.KafkaStreams)
at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
at 
com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
Source)
at com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
at 
com.opentable.chat.service.ChatStorage.handleStreamException(ChatStorage.java:541)
at 
com.opentable.chat.service.ChatStorage$$Lambda$123/149062221.uncaughtException(Unknown
 Source)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1956)

"main" #1 prio=5 os_prio=0 tid=0x7f5608011000 nid=0x3f76 in Object.wait() 
[0x7f5610f04000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1249)
- locked <0xfd302bf0> (a java.lang.Thread)
at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:494)
- locked <0xf171cda8> (a org.apache.kafka.streams.KafkaStreams)
at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:438)
at 
com.opentable.chat.service.ChatStorage$$Lambda$161/1940967023.close(Unknown 
Source)
at com.opentable.chat.service.ChatStorage.closeLog(ChatStorage.java:212)
at com.opentable.chat.service.ChatStorage.close(ChatStorage.java:207)
{code}

Note how the main thread calls close(), which encounters an exception.  It uses 
a StreamThread to dispatch to the handler, which calls close().  Once it tries 
to take the monitor, we are left in a position where main is joined on 
StreamThread-1, but StreamThread-1 is waiting for main to release that monitor.

Arguably it's a bit abusive to call close() in this way (it certainly wasn't 
intentional) -- but to make Kafka Streams robust it should handle any sequence 
of close() invocations in particular gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka-site pull request #47: More edits on 0.10.2 web docs after the release

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka-site/pull/47

More edits on 0.10.2 web docs after the release

Ping @derrickdoo @ewencp for reviews.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka-site 
KMINOR-post-0.10.2-streams

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/47.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #47


commit cab4453c04b48c1ed347ade8ee1c75b6641ea0e3
Author: Guozhang Wang 
Date:   2017-02-22T04:41:06Z

Manual edits needed for 0.10.2 release

Author: Guozhang Wang 
Author: Derrick Or 

Reviewers: Ewen Chesklack-Postava 

Closes #45 from guozhangwang/KRelease-0.10.2

commit 673a1493fd481c3abeb1562c47c5db1ee9905b24
Author: Guozhang Wang 
Date:   2017-02-22T19:00:32Z

post 0.10.2 release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Jun Rao]

Hi, Dong, Jiangjie,

20. (1) I agree that ideally we'd like to use direct RPC for
broker-to-broker communication instead of ZK. However, in the alternative
design, the failed log directory path also serves as the persistent state
for remembering the offline partitions. This is similar to the
controller_managed_state path in your design. The difference is that the
alternative design stores the state in fewer ZK paths, which helps reduce
the controller failover time. (2) I agree that we want the controller to be
the single place to make decisions. However, intuitively, the failure
reporting should be done where the failure is originated. For example, if a
broker fails, the broker reports failure by de-registering from ZK. The
failed log directory path is similar in that regard. (3) I am not worried
about the additional load from extra LeaderAndIsrRequest. What I worry
about is any unnecessary additional complexity in the controller. To me,
the additional complexity in the current design is the additional state
management in each of the existing state handling (e.g., topic creation,
controller failover, etc), and the additional synchronization since the
additional state management is not initiated from the ZK event handling
thread.

21. One of the reasons that we need to send a StopReplicaRequest to offline
replica is to handle controlled shutdown. In that case, a broker is still
alive, but indicates to the controller that it plans to shut down. Being
able to stop the replica in the shutting down broker reduces churns in ISR.
So, for simplicity, it's probably easier to always send a StopReplicaRequest
to any offline replica.

Thanks,

Jun


On Tue, Feb 21, 2017 at 2:37 PM, Dong Lin  wrote:

> Hey Jun,
>
> Thanks much for your comments.
>
> I actually proposed the design to store both offline replicas and created
> replicas in per-broker znode before switching to the design in the current
> KIP. The current design stores created replicas in per-partition znode and
> transmits offline replicas via LeaderAndIsrResponse. The original solution
> is roughly the same as what you suggested. The advantage of the current
> solution is kind of philosophical: 1) we want to transmit data (e.g.
> offline replicas) using RPC and reduce dependency on zookeeper; 2) we want
> controller to be the only one that determines any state (e.g. offline
> replicas) that will be exposed to user. The advantage of the solution to
> store offline replica in zookeeper is that we can save one roundtrip time
> for controller to handle log directory failure. However, this extra
> roundtrip time should not be a big deal since the log directory failure is
> rare and inefficiency of extra latency is less of a problem when there is
> log directory failure.
>
> Do you think the two philosophical advantages of the current KIP make
> sense? If not, then I can switch to the original design that stores offline
> replicas in zookeeper. It is actually written already. One disadvantage is
> that we have to make non-trivial change the KIP (e.g. no create flag in
> LeaderAndIsrRequest and no created flag zookeeper) and restart this KIP
> discussion.
>
> Regarding 21, it seems to me that LeaderAndIsrRequest/StopReplicaRequest
> only makes sense when broker can make the choice (e.g. fetch data for this
> replica or not). In the case that the log directory of the replica is
> already offline, broker have to stop fetching data for this replica
> regardless of what controller tells it to do. Thus it seems cleaner for
> broker to stop fetch data for this replica immediately. The advantage of
> this solution is that the controller logic is simpler since it doesn't need
> to send StopReplicaRequest in case of log directory failure, and the log4j
> log is also cleaner. Is there specific advantage of having controller send
> tells broker to stop fetching data for offline replicas?
>
> Regarding 22, I agree with your observation that it will happen. I will
> update the KIP and specify that broker will exist with proper error message
> in the log and user needs to manually remove partitions and restart the
> broker.
>
> Thanks!
> Dong
>
>
>
> On Mon, Feb 20, 2017 at 10:17 PM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > Sorry for the delay. A few more comments.
> >
> > 20. One complexity that I found in the current KIP is that the way the
> > broker communicates failed replicas to the controller is inefficient.
> When
> > a log directory fails, the broker only sends an indication through ZK to
> > the controller and the controller has to issue a LeaderAndIsrRequest to
> > discover which replicas are offline due to log directory failure. An
> > alternative approach is that when a log directory fails, the broker just
> > writes the failed the directory and the corresponding topic partitions
> in a
> > new failed log directory ZK path like the following.
> >
> > Failed log directory path:
> > 

Re: [DISCUSS] KIP-124: Request rate quotas

[Roger Hoover]

Great to see this KIP and the excellent discussion.

To me, Jun's suggestion makes sense.  If my application is allocated 1
request handler unit, then it's as if I have a Kafka broker with a single
request handler thread dedicated to me.  That's the most I can use, at
least.  That allocation doesn't change even if an admin later increases the
size of the request thread pool on the broker.  It's similar to the CPU
abstraction that VMs and containers get from hypervisors or OS schedulers.
While different client access patterns can use wildly different amounts of
request thread resources per request, a given application will generally
have a stable access pattern and can figure out empirically how many
"request thread units" it needs to meet it's throughput/latency goals.

Cheers,

Roger

On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao  wrote:

> Hi, Rajini,
>
> Thanks for the updated KIP. A few more comments.
>
> 1. A concern of request_time_percent is that it's not an absolute value.
> Let's say you give a user a 10% limit. If the admin doubles the number of
> request handler threads, that user now actually has twice the absolute
> capacity. This may confuse people a bit. So, perhaps setting the quota
> based on an absolute request thread unit is better.
>
> 2. ControlledShutdownRequest is also an inter-broker request and needs to
> be excluded from throttling.
>
> 3. Implementation wise, I am wondering if it's simpler to apply the request
> time throttling first in KafkaApis.handle(). Otherwise, we will need to add
> the throttling logic in each type of request.
>
> Thanks,
>
> Jun
>
> On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram 
> wrote:
>
> > Jun,
> >
> > Thank you for the review.
> >
> > I have reverted to the original KIP that throttles based on request
> handler
> > utilization. At the moment, it uses percentage, but I am happy to change
> to
> > a fraction (out of 1 instead of 100) if required. I have added the
> examples
> > from this discussion to the KIP. Also added a "Future Work" section to
> > address network thread utilization. The configuration is named
> > "request_time_percent" with the expectation that it can also be used as
> the
> > limit for network thread utilization when that is implemented, so that
> > users have to set only one config for the two and not have to worry about
> > the internal distribution of the work between the two thread pools in
> > Kafka.
> >
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao  wrote:
> >
> > > Hi, Rajini,
> > >
> > > Thanks for the proposal.
> > >
> > > The benefit of using the request processing time over the request rate
> is
> > > exactly what people have said. I will just expand that a bit. Consider
> > the
> > > following case. The producer sends a produce request with a 10MB
> message
> > > but compressed to 100KB with gzip. The decompression of the message on
> > the
> > > broker could take 10-15 seconds, during which time, a request handler
> > > thread is completely blocked. In this case, neither the byte-in quota
> nor
> > > the request rate quota may be effective in protecting the broker.
> > Consider
> > > another case. A consumer group starts with 10 instances and later on
> > > switches to 20 instances. The request rate will likely double, but the
> > > actually load on the broker may not double since each fetch request
> only
> > > contains half of the partitions. Request rate quota may not be easy to
> > > configure in this case.
> > >
> > > What we really want is to be able to prevent a client from using too
> much
> > > of the server side resources. In this particular KIP, this resource is
> > the
> > > capacity of the request handler threads. I agree that it may not be
> > > intuitive for the users to determine how to set the right limit.
> However,
> > > this is not completely new and has been done in the container world
> > > already. For example, Linux cgroup (https://access.redhat.com/
> > > documentation/en-US/Red_Hat_Enterprise_Linux/6/html/
> > > Resource_Management_Guide/sec-cpu.html) has the concept of
> > > cpu.cfs_quota_us,
> > > which specifies the total amount of time in microseconds for which all
> > > tasks in a cgroup can run during a one second period. We can
> potentially
> > > model the request handler threads in a similar way. For example, each
> > > request handler thread can be 1 request handler unit and the admin can
> > > configure a limit on how many units (say 0.01) a client can have.
> > >
> > > Regarding not throttling the internal broker to broker requests. We
> could
> > > do that. Alternatively, we could just let the admin configure a high
> > limit
> > > for the kafka user (it may not be able to do that easily based on
> > clientId
> > > though).
> > >
> > > Ideally we want to be able to protect the utilization of the network
> > thread
> > > pool too. The difficult is mostly what Rajini said: (1) The 

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878775#comment-15878775
 ] 

Armin Braun commented on KAFKA-1895:


[~ijuma] maybe you have an opinion on the viability of the above ? :)

> Investigate moving deserialization and decompression out of KafkaConsumer
> -
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and 
> deserializes them into ConsumerRecords which are then handed back as the 
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the 
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as 
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed 
> serialized MemoryRecords chunks and do the deserialization during iteration. 
> This way you could scale this over a thread pool if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Vahid S Hashemian]

+1

Thanks Ewen.

--Vahid



From:   "Ewen Cheslack-Postava" 
To: annou...@apache.org, us...@kafka.apache.org, dev@kafka.apache.org, 
kafka-clie...@googlegroups.com
Date:   02/22/2017 12:33 AM
Subject:[ANNOUNCE] Apache Kafka 0.10.2.0 Released



The Apache Kafka community is pleased to announce the release for Apache
Kafka 0.10.2.0. This is a feature release which includes the completion
of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
requests merged.

All of the changes in this release can be found in the release notes:
https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html

Apache Kafka is a distributed streaming platform with four four core
APIs:

** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output
stream to one or more output topics, effectively transforming the input
streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might capture
every change to a table.three key capabilities:


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react to
the
streams of data.


You can download the source release from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka-0.10.2.0-src.tgz


and binary releases from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz

(experimental 2.12 artifact)

Thanks to the 101 contributors on this release!

Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
Yang Wei, yaojuncn, Yuto Kawamura

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://kafka.apache.org/

Thanks,
Ewen

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Hi Ismael

On point 1,

Sure makes sense will update shortly.

On point 2,

Setter/getter typical to properties/headers api’s traditionally are map styled 
interfaces and what I believe is most expected styled thus the Key, Value 
setter.
Also it would mean rather than an interface, we would be making our internal 
header impl object we have for the array, exposed. E.g. if we had a Map really 
this would be Map.Entry interface, this is the same reasons on the map 
interface I cannot actually make the underlying Node object that’s the 
implementation for HashMap, so that internals can be changed.


On point 3,

I think it people do expect it to be performant, thus why originally concern I 
raised with using an array for to me is an early memory optimisation. I think 
the user experience of properties/headers is on a get/set model. This is why 
its important we have encapsulated logic that then allows us to change this to 
a map, if this becomes and issue, and the memory overhead of hashmap is less so.




On 22/02/2017, 15:56, "isma...@gmail.com on behalf of Ismael Juma" 
 wrote:

Hi all,

Great to see the progress that has been achieved on this one. :) A few
comments regarding the APIs (I'm still reviewing the message format
changes):

1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
named `headers` (we avoid the `get` prefix in Kafka)

2. The `Headers` class is mutable (there's an `add` method). Does it need
to be? If so, it would be good to explain why. Related to that, we should
also explain the thinking around thread-safety. If we keep the `add`
method, it may make sense for it to take a `Header` (that way we can add
things to `Header` without changing the interface).

3. Do we need the `Headers.get()` method? People usually assume that `get`
would be efficient, but depending on the implementation (the current
proposal states that an array would be used), it may not be. If we expect
the number of headers to be small, it doesn't matter though.

Ismael

On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce 
wrote:

> Hi Jason,
>
> Have converted the interface/api bullets into interface code snippets.
>
> Agreed implementation won’t take too long. We have early versions already.
> Maybe a week before you think about merging I would assume it would be 
more
> stabilised? I was thinking then we could fork from your confluent branch,
> making and then holding KIP-82 changes in a patch file, that we can then
> re-fork from apache once KIP98 final is merged, and apply patch with last
> minute changes.
>
> Cheers
> Mike
>
>
> On 22/02/2017, 00:56, "Jason Gustafson"  wrote:
>
> Hey Michael,
>
> Awesome. I have a minor request. The APIs are currently documented as 
a
> wiki list. Would you mind adding a code snippet instead? It's a bit
> easier
> to process.
>
> How will be best to manage this, as we will obviously build off your
> KIP’s
> > protocol changes, to avoid a merge hell, should we branch from your
> branch
> > in the confluent repo or is it worth having a KIP-98 special branch
> in the
> > apache git, that we can branch/fork from?
>
>
> I was thinking about this also. Ideally we'd like to get the changes
> in as
> close together as possible since we only want one magic bump and some
> users
> deploy trunk. The level of effort to change the format for headers
> seems
> not too high. Do you agree? My guess is that the KIP-98 message format
> patch will take 2-3 weeks to review before we merge to trunk, so you
> could
> hold off implementing until that patch has somewhat stabilized. That
> would
> save some potential rebase pain.
>
> -Jason
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to 
others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) 

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Becket Qin]

Thanks Ewen :)

On Wed, Feb 22, 2017 at 5:15 AM, Kenny Gorman  wrote:

> We are excited about this release! Excellent work!
>
> Thanks
> Kenny Gorman
> www.eventador.io
>
> > On Feb 22, 2017, at 2:33 AM, Ewen Cheslack-Postava 
> wrote:
> >
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 0.10.2.0. This is a feature release which includes the completion
> > of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> > requests merged.
> >
> > All of the changes in this release can be found in the release notes:
> > https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> >
> > Apache Kafka is a distributed streaming platform with four four core
> > APIs:
> >
> > ** The Producer API allows an application to publish a stream records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output
> > stream to one or more output topics, effectively transforming the input
> > streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might capture
> > every change to a table.three key capabilities:
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react to
> > the
> > streams of data.
> >
> >
> > You can download the source release from
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka-0.10.2.0-src.tgz
> >
> > and binary releases from
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.11-0.10.2.0.tgz
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.10-0.10.2.0.tgz
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> 0/kafka_2.12-0.10.2.0.tgz
> > (experimental 2.12 artifact)
> >
> > Thanks to the 101 contributors on this release!
> >
> > Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> > Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> > Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> > Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> > Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> > Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> > Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> > Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> > huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> > Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> > Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> > Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> > Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> > Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> > Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> > MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> > Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> > Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> > Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> > Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> > Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> > Yang Wei, yaojuncn, Yuto Kawamura
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > http://kafka.apache.org/
> >
> > Thanks,
> > Ewen
>

Re: [DISCUSS] KIP-124: Request rate quotas

[Jun Rao]

Hi, Rajini,

Thanks for the updated KIP. A few more comments.

1. A concern of request_time_percent is that it's not an absolute value.
Let's say you give a user a 10% limit. If the admin doubles the number of
request handler threads, that user now actually has twice the absolute
capacity. This may confuse people a bit. So, perhaps setting the quota
based on an absolute request thread unit is better.

2. ControlledShutdownRequest is also an inter-broker request and needs to
be excluded from throttling.

3. Implementation wise, I am wondering if it's simpler to apply the request
time throttling first in KafkaApis.handle(). Otherwise, we will need to add
the throttling logic in each type of request.

Thanks,

Jun

On Wed, Feb 22, 2017 at 5:58 AM, Rajini Sivaram 
wrote:

> Jun,
>
> Thank you for the review.
>
> I have reverted to the original KIP that throttles based on request handler
> utilization. At the moment, it uses percentage, but I am happy to change to
> a fraction (out of 1 instead of 100) if required. I have added the examples
> from this discussion to the KIP. Also added a "Future Work" section to
> address network thread utilization. The configuration is named
> "request_time_percent" with the expectation that it can also be used as the
> limit for network thread utilization when that is implemented, so that
> users have to set only one config for the two and not have to worry about
> the internal distribution of the work between the two thread pools in
> Kafka.
>
>
> Regards,
>
> Rajini
>
>
> On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao  wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the proposal.
> >
> > The benefit of using the request processing time over the request rate is
> > exactly what people have said. I will just expand that a bit. Consider
> the
> > following case. The producer sends a produce request with a 10MB message
> > but compressed to 100KB with gzip. The decompression of the message on
> the
> > broker could take 10-15 seconds, during which time, a request handler
> > thread is completely blocked. In this case, neither the byte-in quota nor
> > the request rate quota may be effective in protecting the broker.
> Consider
> > another case. A consumer group starts with 10 instances and later on
> > switches to 20 instances. The request rate will likely double, but the
> > actually load on the broker may not double since each fetch request only
> > contains half of the partitions. Request rate quota may not be easy to
> > configure in this case.
> >
> > What we really want is to be able to prevent a client from using too much
> > of the server side resources. In this particular KIP, this resource is
> the
> > capacity of the request handler threads. I agree that it may not be
> > intuitive for the users to determine how to set the right limit. However,
> > this is not completely new and has been done in the container world
> > already. For example, Linux cgroup (https://access.redhat.com/
> > documentation/en-US/Red_Hat_Enterprise_Linux/6/html/
> > Resource_Management_Guide/sec-cpu.html) has the concept of
> > cpu.cfs_quota_us,
> > which specifies the total amount of time in microseconds for which all
> > tasks in a cgroup can run during a one second period. We can potentially
> > model the request handler threads in a similar way. For example, each
> > request handler thread can be 1 request handler unit and the admin can
> > configure a limit on how many units (say 0.01) a client can have.
> >
> > Regarding not throttling the internal broker to broker requests. We could
> > do that. Alternatively, we could just let the admin configure a high
> limit
> > for the kafka user (it may not be able to do that easily based on
> clientId
> > though).
> >
> > Ideally we want to be able to protect the utilization of the network
> thread
> > pool too. The difficult is mostly what Rajini said: (1) The mechanism for
> > throttling the requests is through Purgatory and we will have to think
> > through how to integrate that into the network layer.  (2) In the network
> > layer, currently we know the user, but not the clientId of the request.
> So,
> > it's a bit tricky to throttle based on clientId there. Plus, the byteOut
> > quota can already protect the network thread utilization for fetch
> > requests. So, if we can't figure out this part right now, just focusing
> on
> > the request handling threads for this KIP is still a useful feature.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 21, 2017 at 4:27 AM, Rajini Sivaram  >
> > wrote:
> >
> > > Thank you all for the feedback.
> > >
> > > Jay: I have removed exemption for consumer heartbeat etc. Agree that
> > > protecting the cluster is more important than protecting individual
> apps.
> > > Have retained the exemption for StopReplicat/LeaderAndIsr etc, these
> are
> > > throttled only if authorization fails (so can't be used for DoS attacks
> > in
> > > a secure 

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

[Ismael Juma]

Great work on the proposal and iterating on it based on community feedback.
As Jun (and others) said, it's likely that minor changes will happen as the
PR is reviewed and additional testing takes place since this is a
significant change.

I am +1 (binding) on the proposal without optional keys and values to keep
things consistent. If we show during performance testing that this is
worthwhile, we can update the proposal.

Ismael

On Tue, Feb 21, 2017 at 6:23 PM, Jun Rao  wrote:

> It seems that it's simpler and more consistent to avoid optional keys and
> values. Not sure if it's worth squeezing every byte at the expense of
> additional complexity. Other than that, +1 from me.
>
> Also, since this is a large KIP, minor changes may arise as we start the
> implementation. It would be good if we can keep the community posted of
> those changes, if any.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 21, 2017 at 4:33 PM, Michael Pearce 
> wrote:
>
> > If the argument and objective within this KIP is to keep the overhead of
> > the protocol as small as possible and remove redundancy, and every byte
> is
> > being counted and the introduction of varInts, then it would make sense
> to
> > use attributes to me.
> >
> >
> > On 22/02/2017, 00:14, "Jason Gustafson"  wrote:
> >
> > Done. I've left the key and value as optional since we may not have
> > reached
> > consensus on whether to use attributes or not. Perhaps we should just
> > keep
> > it simple and not do it? The benefit seems small.
> >
> > -Jason
> >
> > On Tue, Feb 21, 2017 at 4:05 PM, Michael Pearce <
> michael.pea...@ig.com
> > >
> > wrote:
> >
> > > Ok, no worries, can you add it back ValueLen on this KIP, and
> update
> > the
> > > doc, then we can work from that ☺
> > >
> > > Cheers
> > > Mike
> > >
> > > On 22/02/2017, 00:02, "Jason Gustafson" 
> wrote:
> > >
> > > I feel it was a little odd to leave out the value length
> anyway,
> > so I
> > > would
> > > rather add it back and put headers at the end. This is more
> > consistent
> > > with
> > > the rest of the Kafka protocol.
> > >
> > > -Jason
> > >
> > > On Tue, Feb 21, 2017 at 3:58 PM, Michael Pearce <
> > michael.pea...@ig.com
> > > >
> > > wrote:
> > >
> > > > Or we keep as is (valuelen removed), and headers are added
> with
> > > headers
> > > > length..
> > > >
> > > > On 21/02/2017, 23:38, "Apurva Mehta" 
> > wrote:
> > > >
> > > > Right now, we don't need the value length: since it is
> the
> > last
> > > item
> > > > in the
> > > > message, and we have the message length, we can deduce
> the
> > value
> > > > length.
> > > > However, if we are adding record headers to the end, we
> > would
> > > need to
> > > > introduce the value length along with that change.
> > > >
> > > > On Tue, Feb 21, 2017 at 3:34 PM, Michael Pearce <
> > > michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > It seems I cannot add comment on the doc.
> > > > >
> > > > > In the section around the message protocol.
> > > > >
> > > > > It has stated:
> > > > >
> > > > > Message =>
> > > > > Length => uintVar
> > > > > Attributes => int8
> > > > > TimestampDelta => intVar
> > > > > OffsetDelta => uintVar
> > > > > KeyLen => uintVar [OPTIONAL]
> > > > > Key => data [OPTIONAL]
> > > > > Value => data [OPTIONAL]
> > > > >
> > > > > Should it not be: (added missing value len)
> > > > >
> > > > > Message =>
> > > > > Length => uintVar
> > > > > Attributes => int8
> > > > > TimestampDelta => intVar
> > > > > OffsetDelta => uintVar
> > > > > KeyLen => uintVar [OPTIONAL]
> > > > > Key => data [OPTIONAL]
> > > > > ValueLen => uintVar [OPTIONAL]
> > > > > Value => data [OPTIONAL]
> > > > >
> > > > >
> > > > >
> > > > > On 21/02/2017, 23:07, "Joel Koshy" <
> jjkosh...@gmail.com>
> > > wrote:
> > > > >
> > > > > I left a couple of comments/questions directly on
> the
> > > google-doc
> > > > >  > > > > GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8>
> > > > > - I found it much more tractable for a proposal of
> > this
> > > size to
> > > > > discuss in
> > > > > context within the doc. The permissions on the doc
> > don't
> > > let
> 

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

[Jason Gustafson]

Thanks Jun, let's go with the simple approach (not using attributes) and
revise later if we find a compelling reason. I'll update the docs.

-Jason

On Tue, Feb 21, 2017 at 6:23 PM, Jun Rao  wrote:

> It seems that it's simpler and more consistent to avoid optional keys and
> values. Not sure if it's worth squeezing every byte at the expense of
> additional complexity. Other than that, +1 from me.
>
> Also, since this is a large KIP, minor changes may arise as we start the
> implementation. It would be good if we can keep the community posted of
> those changes, if any.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 21, 2017 at 4:33 PM, Michael Pearce 
> wrote:
>
> > If the argument and objective within this KIP is to keep the overhead of
> > the protocol as small as possible and remove redundancy, and every byte
> is
> > being counted and the introduction of varInts, then it would make sense
> to
> > use attributes to me.
> >
> >
> > On 22/02/2017, 00:14, "Jason Gustafson"  wrote:
> >
> > Done. I've left the key and value as optional since we may not have
> > reached
> > consensus on whether to use attributes or not. Perhaps we should just
> > keep
> > it simple and not do it? The benefit seems small.
> >
> > -Jason
> >
> > On Tue, Feb 21, 2017 at 4:05 PM, Michael Pearce <
> michael.pea...@ig.com
> > >
> > wrote:
> >
> > > Ok, no worries, can you add it back ValueLen on this KIP, and
> update
> > the
> > > doc, then we can work from that ☺
> > >
> > > Cheers
> > > Mike
> > >
> > > On 22/02/2017, 00:02, "Jason Gustafson" 
> wrote:
> > >
> > > I feel it was a little odd to leave out the value length
> anyway,
> > so I
> > > would
> > > rather add it back and put headers at the end. This is more
> > consistent
> > > with
> > > the rest of the Kafka protocol.
> > >
> > > -Jason
> > >
> > > On Tue, Feb 21, 2017 at 3:58 PM, Michael Pearce <
> > michael.pea...@ig.com
> > > >
> > > wrote:
> > >
> > > > Or we keep as is (valuelen removed), and headers are added
> with
> > > headers
> > > > length..
> > > >
> > > > On 21/02/2017, 23:38, "Apurva Mehta" 
> > wrote:
> > > >
> > > > Right now, we don't need the value length: since it is
> the
> > last
> > > item
> > > > in the
> > > > message, and we have the message length, we can deduce
> the
> > value
> > > > length.
> > > > However, if we are adding record headers to the end, we
> > would
> > > need to
> > > > introduce the value length along with that change.
> > > >
> > > > On Tue, Feb 21, 2017 at 3:34 PM, Michael Pearce <
> > > michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > It seems I cannot add comment on the doc.
> > > > >
> > > > > In the section around the message protocol.
> > > > >
> > > > > It has stated:
> > > > >
> > > > > Message =>
> > > > > Length => uintVar
> > > > > Attributes => int8
> > > > > TimestampDelta => intVar
> > > > > OffsetDelta => uintVar
> > > > > KeyLen => uintVar [OPTIONAL]
> > > > > Key => data [OPTIONAL]
> > > > > Value => data [OPTIONAL]
> > > > >
> > > > > Should it not be: (added missing value len)
> > > > >
> > > > > Message =>
> > > > > Length => uintVar
> > > > > Attributes => int8
> > > > > TimestampDelta => intVar
> > > > > OffsetDelta => uintVar
> > > > > KeyLen => uintVar [OPTIONAL]
> > > > > Key => data [OPTIONAL]
> > > > > ValueLen => uintVar [OPTIONAL]
> > > > > Value => data [OPTIONAL]
> > > > >
> > > > >
> > > > >
> > > > > On 21/02/2017, 23:07, "Joel Koshy" <
> jjkosh...@gmail.com>
> > > wrote:
> > > > >
> > > > > I left a couple of comments/questions directly on
> the
> > > google-doc
> > > > >  > > > > GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8>
> > > > > - I found it much more tractable for a proposal of
> > this
> > > size to
> > > > > discuss in
> > > > > context within the doc. The permissions on the doc
> > don't
> > > let
> > > > everyone
> > > > > view
> > > > > comments, so if there are any material changes that
> > come
> > > out of
> > > > the
> > > > > discussions in those comment threads we can
> > summarize here.
> > > > 

[jira] [Commented] (KAFKA-4786) Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878599#comment-15878599
 ] 

ASF GitHub Bot commented on KAFKA-4786:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2586

KAFKA-4786: Wait for heartbeat thread to terminate in consumer close



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4786

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2586


commit 01a9bc2819ea9f4949ea2baa7ab46b6c5708424f
Author: Rajini Sivaram 
Date:   2017-02-22T15:40:19Z

KAFKA-4786: Wait for heartbeat thread to terminate in consumer close




> Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose
> 
>
> Key: KAFKA-4786
> URL: https://issues.apache.org/jira/browse/KAFKA-4786
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>
> I've seen this failure in Jenkins a number of times:
> {code}
> java.lang.AssertionError: Heartbeat thread active after close
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertFalse(Assert.java:64)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testHeartbeatThreadClose(ConsumerCoordinatorTest.java:1425)
> {code}
> https://jenkins.confluent.io/job/kafka-0.10.2/88/testReport/org.apache.kafka.clients.consumer.internals/ConsumerCoordinatorTest/testHeartbeatThreadClose/
> [~rsivaram], do you think you can take a look since you wrote the test?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2586: KAFKA-4786: Wait for heartbeat thread to terminate...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2586

KAFKA-4786: Wait for heartbeat thread to terminate in consumer close



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4786

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2586


commit 01a9bc2819ea9f4949ea2baa7ab46b6c5708424f
Author: Rajini Sivaram 
Date:   2017-02-22T15:40:19Z

KAFKA-4786: Wait for heartbeat thread to terminate in consumer close




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878549#comment-15878549
 ] 

Armin Braun commented on KAFKA-4569:


[~ijuma][~hachikuji] it looks like there was an attemp at this already in the 
code the test starts with this:

{code}
int rebalanceTimeoutMs = 6;
int sessionTimeoutMs = 3;
int heartbeatIntervalMs = 3000;

// adjust auto commit interval lower than heartbeat so we don't need to 
deal with
// a concurrent heartbeat request
int autoCommitIntervalMs = 1000;

{code}

Maybe the interval was used as an initial delay on the heartbeat thread action 
before and that got turned into starting on a heartbeat instead of a wait 
accidentally at some point?

> Transient failure in 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable
> -
>
> Key: KAFKA-4569
> URL: https://issues.apache.org/jira/browse/KAFKA-4569
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
> Fix For: 0.10.3.0
>
>
> One example is:
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/370/testReport/junit/org.apache.kafka.clients.consumer/KafkaConsumerTest/testWakeupWithFetchDataAvailable/
> {code}
> Stacktrace
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.fail(Assert.java:95)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable(KafkaConsumerTest.java:679)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Ismael Juma]

Hi all,

Great to see the progress that has been achieved on this one. :) A few
comments regarding the APIs (I'm still reviewing the message format
changes):

1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
named `headers` (we avoid the `get` prefix in Kafka)

2. The `Headers` class is mutable (there's an `add` method). Does it need
to be? If so, it would be good to explain why. Related to that, we should
also explain the thinking around thread-safety. If we keep the `add`
method, it may make sense for it to take a `Header` (that way we can add
things to `Header` without changing the interface).

3. Do we need the `Headers.get()` method? People usually assume that `get`
would be efficient, but depending on the implementation (the current
proposal states that an array would be used), it may not be. If we expect
the number of headers to be small, it doesn't matter though.

Ismael

On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce 
wrote:

> Hi Jason,
>
> Have converted the interface/api bullets into interface code snippets.
>
> Agreed implementation won’t take too long. We have early versions already.
> Maybe a week before you think about merging I would assume it would be more
> stabilised? I was thinking then we could fork from your confluent branch,
> making and then holding KIP-82 changes in a patch file, that we can then
> re-fork from apache once KIP98 final is merged, and apply patch with last
> minute changes.
>
> Cheers
> Mike
>
>
> On 22/02/2017, 00:56, "Jason Gustafson"  wrote:
>
> Hey Michael,
>
> Awesome. I have a minor request. The APIs are currently documented as a
> wiki list. Would you mind adding a code snippet instead? It's a bit
> easier
> to process.
>
> How will be best to manage this, as we will obviously build off your
> KIP’s
> > protocol changes, to avoid a merge hell, should we branch from your
> branch
> > in the confluent repo or is it worth having a KIP-98 special branch
> in the
> > apache git, that we can branch/fork from?
>
>
> I was thinking about this also. Ideally we'd like to get the changes
> in as
> close together as possible since we only want one magic bump and some
> users
> deploy trunk. The level of effort to change the format for headers
> seems
> not too high. Do you agree? My guess is that the KIP-98 message format
> patch will take 2-3 weeks to review before we merge to trunk, so you
> could
> hold off implementing until that patch has somewhat stabilized. That
> would
> save some potential rebase pain.
>
> -Jason
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Kenny Gorman]

We are excited about this release! Excellent work!

Thanks
Kenny Gorman
www.eventador.io

> On Feb 22, 2017, at 2:33 AM, Ewen Cheslack-Postava  wrote:
> 
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.2.0. This is a feature release which includes the completion
> of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> requests merged.
> 
> All of the changes in this release can be found in the release notes:
> https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> 
> Apache Kafka is a distributed streaming platform with four four core
> APIs:
> 
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
> 
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
> 
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
> 
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
> 
> 
> With these APIs, Kafka can be used for two broad classes of application:
> 
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
> 
> ** Building real-time streaming applications that transform or react to
> the
> streams of data.
> 
> 
> You can download the source release from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka-0.10.2.0-src.tgz
> 
> and binary releases from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
> (experimental 2.12 artifact)
> 
> Thanks to the 101 contributors on this release!
> 
> Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> Yang Wei, yaojuncn, Yuto Kawamura
> 
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
> 
> Thanks,
> Ewen

[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection

[Dipen Patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878366#comment-15878366
 ] 

Dipen Patel commented on KAFKA-1935:


Thanks for the clarifications. If this isn't a good issue for me to be working 
on at the moment, I can definitely find another issue to get myself acquainted 
with the codebase.   

> Consumer should use a separate socket for Coordinator connection
> 
>
> Key: KAFKA-1935
> URL: https://issues.apache.org/jira/browse/KAFKA-1935
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>  Labels: newbie
>
> KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be 
> able to create separate sockets for the same server for coordinator / broker 
> roles.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878251#comment-15878251
 ] 

Ismael Juma commented on KAFKA-4569:


[~hachikuji], any thoughts on how we should fix this now that 
[~original-brownbear] has narrowed it down?

> Transient failure in 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable
> -
>
> Key: KAFKA-4569
> URL: https://issues.apache.org/jira/browse/KAFKA-4569
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
> Fix For: 0.10.3.0
>
>
> One example is:
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/370/testReport/junit/org.apache.kafka.clients.consumer/KafkaConsumerTest/testWakeupWithFetchDataAvailable/
> {code}
> Stacktrace
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.fail(Assert.java:95)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable(KafkaConsumerTest.java:679)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> 

Re: [DISCUSS] KIP-124: Request rate quotas

[Rajini Sivaram]

Jun,

Thank you for the review.

I have reverted to the original KIP that throttles based on request handler
utilization. At the moment, it uses percentage, but I am happy to change to
a fraction (out of 1 instead of 100) if required. I have added the examples
from this discussion to the KIP. Also added a "Future Work" section to
address network thread utilization. The configuration is named
"request_time_percent" with the expectation that it can also be used as the
limit for network thread utilization when that is implemented, so that
users have to set only one config for the two and not have to worry about
the internal distribution of the work between the two thread pools in Kafka.


Regards,

Rajini


On Wed, Feb 22, 2017 at 12:23 AM, Jun Rao  wrote:

> Hi, Rajini,
>
> Thanks for the proposal.
>
> The benefit of using the request processing time over the request rate is
> exactly what people have said. I will just expand that a bit. Consider the
> following case. The producer sends a produce request with a 10MB message
> but compressed to 100KB with gzip. The decompression of the message on the
> broker could take 10-15 seconds, during which time, a request handler
> thread is completely blocked. In this case, neither the byte-in quota nor
> the request rate quota may be effective in protecting the broker. Consider
> another case. A consumer group starts with 10 instances and later on
> switches to 20 instances. The request rate will likely double, but the
> actually load on the broker may not double since each fetch request only
> contains half of the partitions. Request rate quota may not be easy to
> configure in this case.
>
> What we really want is to be able to prevent a client from using too much
> of the server side resources. In this particular KIP, this resource is the
> capacity of the request handler threads. I agree that it may not be
> intuitive for the users to determine how to set the right limit. However,
> this is not completely new and has been done in the container world
> already. For example, Linux cgroup (https://access.redhat.com/
> documentation/en-US/Red_Hat_Enterprise_Linux/6/html/
> Resource_Management_Guide/sec-cpu.html) has the concept of
> cpu.cfs_quota_us,
> which specifies the total amount of time in microseconds for which all
> tasks in a cgroup can run during a one second period. We can potentially
> model the request handler threads in a similar way. For example, each
> request handler thread can be 1 request handler unit and the admin can
> configure a limit on how many units (say 0.01) a client can have.
>
> Regarding not throttling the internal broker to broker requests. We could
> do that. Alternatively, we could just let the admin configure a high limit
> for the kafka user (it may not be able to do that easily based on clientId
> though).
>
> Ideally we want to be able to protect the utilization of the network thread
> pool too. The difficult is mostly what Rajini said: (1) The mechanism for
> throttling the requests is through Purgatory and we will have to think
> through how to integrate that into the network layer.  (2) In the network
> layer, currently we know the user, but not the clientId of the request. So,
> it's a bit tricky to throttle based on clientId there. Plus, the byteOut
> quota can already protect the network thread utilization for fetch
> requests. So, if we can't figure out this part right now, just focusing on
> the request handling threads for this KIP is still a useful feature.
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 21, 2017 at 4:27 AM, Rajini Sivaram 
> wrote:
>
> > Thank you all for the feedback.
> >
> > Jay: I have removed exemption for consumer heartbeat etc. Agree that
> > protecting the cluster is more important than protecting individual apps.
> > Have retained the exemption for StopReplicat/LeaderAndIsr etc, these are
> > throttled only if authorization fails (so can't be used for DoS attacks
> in
> > a secure cluster, but allows inter-broker requests to complete without
> > delays).
> >
> > I will wait another day to see if these is any objection to quotas based
> on
> > request processing time (as opposed to request rate) and if there are no
> > objections, I will revert to the original proposal with some changes.
> >
> > The original proposal was only including the time used by the request
> > handler threads (that made calculation easy). I think the suggestion is
> to
> > include the time spent in the network threads as well since that may be
> > significant. As Jay pointed out, it is more complicated to calculate the
> > total available CPU time and convert to a ratio when there *m* I/O
> threads
> > and *n* network threads. ThreadMXBean#getThreadCPUTime() may give us
> what
> > we want, but it can be very expensive on some platforms. As Becket and
> > Guozhang have pointed out, we do have several time measurements already
> for
> > generating metrics that we could use, though we 

[GitHub] kafka pull request #2585: MINOR: Fix potential integer overflow and String.f...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2585

MINOR: Fix potential integer overflow and String.format issue



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka overflow-and-format-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2585


commit 5be2d9867dad7a6d844494cc3cadf4855a97d56a
Author: Ismael Juma 
Date:   2017-02-22T13:29:14Z

MINOR: Fix potential integer overflow and String.format issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

[Tianji Li]

Hi Florian,

Just curious, what 'shared storage' you guys use to keep the files before
ingested into Kafka?

In our case, we could not figure out such a nice distributed+shared file
system that is NOT HDFS alike and runs before Kafka. So we use individual
harddisks on connector machines and keep offsets/etc local on the
harddisks. That is, before Kafka Connect, everything is disconnected, and
from Kafka Connect, everything is on a distributed system.

Thanks
Tianji


On Mon, Feb 20, 2017 at 6:24 AM, Florian Hussonnois 
wrote:

> Hi Jason,
>
> Yes, this is the idea. The connector assigns a subset of files to each
> task.
>
> A task stores the size of file, the bytes offset and the bytes size of the
> last sent record as a source offsets.
> A file is finished when recordBytesOffsets + recordBytesSize =
> fileBytesSize.
>
> The connector should be able to start a thread in background to track
> offsets for each assigned file.
> When all tasks has finished the connector can stop tasks or assigned new
> files by requesting tasks reconfiguration.
>
> Another advantage of monitoring source offsets from the connector is detect
> slow or failed tasks and if necessary to be able to restart all tasks.
>
> Thanks,
>
> 2017-02-18 6:47 GMT+01:00 Jason Gustafson :
>
> > Hey Florian,
> >
> > Can you explain a bit more how having access to the offset storage from
> the
> > connector helps in your use case? I guess you are planning to use offsets
> > to be able to tell when a task has finished a file?
> >
> > Thanks,
> > Jason
> >
> > On Fri, Feb 17, 2017 at 4:45 AM, Florian Hussonnois <
> fhussonn...@gmail.com
> > >
> > wrote:
> >
> > > Hi Kafka Team,
> > >
> > > I'm developping a connector which need to monitor the progress of its
> > tasks
> > > in order to be able to request a tasks reconfiguration in some
> > situations.
> > >
> > > Our connector is pretty simple. It's used to stream a thousands of
> files
> > > into Kafka. The connector scans directories then schedules each task
> > with a
> > > set of assigned files.
> > > When tasks are no longer required or new files are detected the
> connector
> > > requests a reconfiguration.
> > >
> > > In addition, files are store into a shared storage which is accessible
> > from
> > > each connect worker. In that way, we can distribute file streaming.
> > >
> > > For that prupose, it would be very convenient to have access to an
> > > offsetStorageReader instance from either the Connector class or the
> > > ConnectorContext class.
> > >
> > > I found a similar question:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg50579.html
> > >
> > > Do you think this improvement could be considered ? I can contribute to
> > it.
> > >
> > > Thanks,
> > >
> > > --
> > > Florian HUSSONNOIS
> > >
> >
>
>
>
> --
> Florian HUSSONNOIS
>

[jira] [Comment Edited] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

[Andrew Olson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878152#comment-15878152
 ] 

Andrew Olson edited comment on KAFKA-3866 at 2/22/17 1:06 PM:
--

We saw the following error when setting a very low expiration time (-maxlife 
"30 seconds" and -maxrenewlife "90 seconds"),

{noformat}ERROR NextRefresh: Tue Feb 21 12:42:59 CST 2017 is in the past: 
exiting refresh
thread. Check clock sync between this host and KDC - (KDC's clock is likely 
ahead of this host). 
Manual intervention will be required for this client to successfully 
authenticate. Exiting refresh 
thread. (org.apache.kafka.common.security.kerberos.KerberosLogin){noformat}

Would this change fix that scenario?


was (Author: noslowerdna):
We saw the following error when setting a very low expiration time (-maxlife 
"30 seconds" and -maxrenewlife "90 seconds"),

{noformat}ERROR NextRefresh: Tue Feb 21 12:42:59 CST 2017 is in the past: 
exiting refresh thread. Check clock sync between this host and KDC - (KDC's 
clock is likely ahead of this host). Manual intervention will be required for 
this client to successfully authenticate. Exiting refresh thread. 
(org.apache.kafka.common.security.kerberos.KerberosLogin){noformat}

Would this change fix that scenario?

> KerberosLogin refresh time bug and other improvements
> -
>
> Key: KAFKA-3866
> URL: https://issues.apache.org/jira/browse/KAFKA-3866
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> ZOOKEEPER-2295 describes a bug in the Kerberos refresh time logic that is 
> also present in our KerberosLogin class. While looking at the code, I found a 
> number of things that could be improved. More details in the PR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

[Andrew Olson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878152#comment-15878152
 ] 

Andrew Olson commented on KAFKA-3866:
-

We saw the following error when setting a very low expiration time (-maxlife 
"30 seconds" and -maxrenewlife "90 seconds"),

{noformat}ERROR NextRefresh: Tue Feb 21 12:42:59 CST 2017 is in the past: 
exiting refresh thread. Check clock sync between this host and KDC - (KDC's 
clock is likely ahead of this host). Manual intervention will be required for 
this client to successfully authenticate. Exiting refresh thread. 
(org.apache.kafka.common.security.kerberos.KerberosLogin){noformat}

Would this change fix that scenario?

> KerberosLogin refresh time bug and other improvements
> -
>
> Key: KAFKA-3866
> URL: https://issues.apache.org/jira/browse/KAFKA-3866
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> ZOOKEEPER-2295 describes a bug in the Kerberos refresh time logic that is 
> also present in our KerberosLogin class. While looking at the code, I found a 
> number of things that could be improved. More details in the PR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #1958

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-4703) Test with two SASL_SSL listeners with different JAAS contexts

[Balint Molnar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balint Molnar updated KAFKA-4703:
-
Description: 
[~rsivaram] suggested the following in https://github.com/apache/kafka/pull/2406

{quote}
I think this feature allows two SASL_SSL listeners, one for external and one 
for internal and the two can use different mechanisms and different JAAS 
contexts. That makes the multi-mechanism configuration neater. I think it will 
be useful to have an integration test for this, perhaps change 
SaslMultiMechanismConsumerTest.
{quote}

And my reply:

{quote}
I think it's a bit tricky to support multiple listeners in 
KafkaServerTestHarness. Maybe it's easier to do the test you suggest in 
MultipleListenersWithSameSecurityProtocolTest.
{quote}

  was:
[~rsivaram] suggested the following in 
https://github.com/apache/kafka/pull/2406:

{quote}
I think this feature allows two SASL_SSL listeners, one for external and one 
for internal and the two can use different mechanisms and different JAAS 
contexts. That makes the multi-mechanism configuration neater. I think it will 
be useful to have an integration test for this, perhaps change 
SaslMultiMechanismConsumerTest.
{quote}

And my reply:

{quote}
I think it's a bit tricky to support multiple listeners in 
KafkaServerTestHarness. Maybe it's easier to do the test you suggest in 
MultipleListenersWithSameSecurityProtocolTest.
{quote}


> Test with two SASL_SSL listeners with different JAAS contexts
> -
>
> Key: KAFKA-4703
> URL: https://issues.apache.org/jira/browse/KAFKA-4703
> Project: Kafka
>  Issue Type: Test
>Reporter: Ismael Juma
>Assignee: Balint Molnar
>  Labels: newbie
>
> [~rsivaram] suggested the following in 
> https://github.com/apache/kafka/pull/2406
> {quote}
> I think this feature allows two SASL_SSL listeners, one for external and one 
> for internal and the two can use different mechanisms and different JAAS 
> contexts. That makes the multi-mechanism configuration neater. I think it 
> will be useful to have an integration test for this, perhaps change 
> SaslMultiMechanismConsumerTest.
> {quote}
> And my reply:
> {quote}
> I think it's a bit tricky to support multiple listeners in 
> KafkaServerTestHarness. Maybe it's easier to do the test you suggest in 
> MultipleListenersWithSameSecurityProtocolTest.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1292

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-4708; Fix transient failure in

--
[...truncated 156.87 KB...]

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 

[GitHub] kafka pull request #2584: MINOR: Fix transient failure of testCannotSendToIn...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2584

MINOR: Fix transient failure of testCannotSendToInternalTopic

It’s a simple matter of creating the internal topic before trying to send
to it. Otherwise, we could get an `UnknownTopicOrPartitionException`
in some cases.

Without the change, I could reproduce a failure in less than
5 runs. With the change, 30 consecutive runs succeeded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
test-cannot-send-to-internal-topic-transient-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2584


commit a6a1bf20f7ceee1c5f1f51ceef64c52830dc3df7
Author: Ismael Juma 
Date:   2017-02-22T11:52:18Z

MINOR: Fix transient failure of testCannotSendToInternalTopic

It’s a simple matter of creating the internal topic before trying to send
to it. Otherwise, we could get an `UnknownTopicOrPartitionException`
in some cases.

Without the change, I could reproduce a failure in less than
5 runs. With the change, 30 consecutive runs succeeded.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-0.10.2-jdk7 #85

[Apache Jenkins Server]

See 


Changes:

[ismael] HOTFIX: Fix version in __init__.py

--
[...truncated 322.80 KB...]

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle PASSED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments STARTED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion STARTED

kafka.server.OffsetCommitTest > 

[jira] [Updated] (KAFKA-4708) Fix Transient failure in BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4708:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Fix Transient failure in 
> BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput
> 
>
> Key: KAFKA-4708
> URL: https://issues.apache.org/jira/browse/KAFKA-4708
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Colin P. McCabe
> Fix For: 0.10.3.0
>
>
> {code}
> kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
> FAILED
> org.junit.ComparisonFailure: expected:<[localhost:34091 (id: 0 rack: 
> null) -> (]> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> kafka.admin.BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput(BrokerApiVersionsCommandTest.scala:44)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4195) support throttling on request rate

[Charly Molter (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878008#comment-15878008
 ] 

Charly Molter commented on KAFKA-4195:
--

For those looking for the KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas


> support throttling on request rate
> --
>
> Key: KAFKA-4195
> URL: https://issues.apache.org/jira/browse/KAFKA-4195
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
>  Labels: needs-kip
>
> Currently, we can throttle the client by data volume. However, if a client 
> sends requests too quickly (e.g., a consumer with min.byte configured to 0), 
> it can still overwhelm the broker. It would be useful to additionally support 
> throttling by request rate. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)