Re: [DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Jeff Klukas]

I've updated the KIP to use the topics.regex name and opened a WIP PR with
an implementation that shows some additional complexity in how the
configuration option gets passed through, affecting various public function
signatures.

I would appreciate any eyes on that for feedback on whether more design
discussion needs to happen in the KIP.

https://github.com/apache/kafka/pull/4151

On Fri, Oct 27, 2017 at 7:50 AM, Jeff Klukas  wrote:

> I added a note in the KIP about ConfigException being thrown. I also
> changed the proposed default for the new config to empty string rather than
> null.
>
> Absent a clear definition of what "common" regex syntax is, it seems an
> undue burden to ask the user to guess at what Pattern features are safe. If
> we do end up implementing a different regex style, I think it will be
> necessary to still support the Java Pattern style long-term as an option.
> If we want to use a different regex style as default down the road, we
> could require "power users" of Java Pattern to enable an additional config
> option to maintain compatibility.
>
> One additional change I might make to the KIP is that 'topics.regex' might
> be a better choice for config name than 'topics.pattern'. That would be in
> keeping with RegexRouter that has a 'regex' configuration option rather
> than 'pattern'.
>
> On Thu, Oct 26, 2017 at 11:00 PM, Ewen Cheslack-Postava  > wrote:
>
>> It's fine to be more detailed, but ConfigException is already implied for
>> all other config issues as well.
>>
>> Default could be either null or just empty string. re: alternatives, if
>> you
>> wanted to be slightly more detailed (though still a bit vague) re:
>> supported syntax, you could just say that while Pattern is used, we only
>> guarantee support for common regular expression syntax. Not sure if
>> there's
>> a good way of defining what "common" syntax is.
>>
>> Otherwise LGTM, and thanks for helping fill in a longstanding gap!
>>
>> -Ewen
>>
>> On Thu, Oct 26, 2017 at 7:56 PM, Ted Yu  wrote:
>>
>> > bq. Users may specify only one of 'topics' or 'topics.pattern'.
>> >
>> > Can you fill in which exception would be thrown if both of them are
>> > specified
>> > ?
>> >
>> > Cheers
>> >
>> > On Thu, Oct 26, 2017 at 6:27 PM, Jeff Klukas  wrote:
>> >
>> > > Looking for feedback on
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 215%3A+Add+topic+regex+support+for+Connect+sinks
>> > >
>> >
>>
>
>

[GitHub] kafka pull request #4151: [WIP] KIP-215: Add topic regex support for Connect...

[jklukas]

GitHub user jklukas opened a pull request:

https://github.com/apache/kafka/pull/4151

[WIP] KIP-215: Add topic regex support for Connect sinks

I'm starting this WIP PR to prove out the concept of the KIP and identify 
some additional public interface changes that I had not previously anticipated 
would be necessary.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jklukas/kafka connect-topics.regex

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4151


commit cb8e2615e9ef1bf78e3d44671123b7e7231ee5f6
Author: Jeff Klukas 
Date:   2017-10-28T01:10:49Z

Add topics.regex option and ensure existing tests pass




---

[GitHub] kafka pull request #4150: MINOR: Add valid values for message.timestamp.type

[makearl]

GitHub user makearl opened a pull request:

https://github.com/apache/kafka/pull/4150

MINOR: Add valid values for message.timestamp.type

The documentation for `message.timestamp.type` is missing valid values 
(https://kafka.apache.org/documentation/#topicconfigs). This change adds valid 
values for that config

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/makearl/kafka 
add-topic-message-timestamp-defaults

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4150


commit 5f314b10bf9eb464a2897fa97e9d96490c83
Author: makearl 
Date:   2017-10-28T00:47:47Z

Add valid values for message.timestamp.type




---

Jenkins build is back to normal : kafka-0.11.0-jdk7 #331

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Matt Farmer]

I can’t think of a reason that would be problematic.

Most of the time I would write a handler like this, I either want to ignore
the error or fail and bring everything down so that I can spin it back up
later and resume from earlier offsets. When we start up after crashing
we’ll eventually try to process the message we failed to produce again.

I’m concerned that “putting in a queue for later” opens you up to putting
messages into the destination topic in an unexpected order. However if
others feel differently, I’m happy to talk about it.

On Fri, Oct 27, 2017 at 7:10 PM Guozhang Wang  wrote:

> > Please correct me if I'm wrong, but my understanding is that the record
> > metadata is always null if an exception occurred while trying to produce.
>
> That is right. Thanks.
>
> I looked at the example code, and one thing I realized that since we are
> not passing the context in the handle function, we may not be implement the
> logic to send the fail records into another queue for future processing.
> Would people think that would be a big issue?
>
>
> Guozhang
>
>
> On Thu, Oct 26, 2017 at 12:14 PM, Matt Farmer  wrote:
>
> > Hello all,
> >
> > I've updated the KIP based on this conversation, and made it so that its
> > interface, config setting, and parameters line up more closely with the
> > interface in KIP-161 (deserialization handler).
> >
> > I believe there are a few specific questions I need to reply to.
> >
> > > The question I had about then handle parameters are around the record,
> > > should it be `ProducerRecord`, or be generics of
> > > `ProducerRecord` or `ProducerRecord > > Object, ? extends Object>`?
> >
> > At this point in the code we're guaranteed that this is a
> > ProducerRecord, so the generics would just make it harder
> > to work with the key and value.
> >
> > > Also, should the handle function include the `RecordMetadata` as well
> in
> > > case it is not null?
> >
> > Please correct me if I'm wrong, but my understanding is that the record
> > metadata is always null if an exception occurred while trying to produce.
> >
> > > We may probably try to write down at least the following handling logic
> > and
> > > see if the given API is sufficient for it
> >
> > I've added some examples to the KIP. Let me know what you think.
> >
> > Cheers,
> > Matt
> >
> > On Mon, Oct 23, 2017 at 9:00 PM Matt Farmer  wrote:
> >
> > > Thanks for this feedback. I’m at a conference right now and am planning
> > on
> > > updating the KIP again with details from this conversation later this
> > week.
> > >
> > > I’ll shoot you a more detailed response then! :)
> > > On Mon, Oct 23, 2017 at 8:16 PM Guozhang Wang 
> > wrote:
> > >
> > >> Thanks for the KIP Matt.
> > >>
> > >> Regarding the handle interface of ProductionExceptionHandlerResponse,
> > >> could
> > >> you write it on the wiki also, along with the actual added config
> names
> > >> (e.g. what
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+
> > deserialization+exception+handlers
> > >> described).
> > >>
> > >> The question I had about then handle parameters are around the record,
> > >> should it be `ProducerRecord`, or be generics of
> > >> `ProducerRecord` or `ProducerRecord extends
> > >> Object, ? extends Object>`?
> > >>
> > >> Also, should the handle function include the `RecordMetadata` as well
> in
> > >> case it is not null?
> > >>
> > >> We may probably try to write down at least the following handling
> logic
> > >> and
> > >> see if the given API is sufficient for it: 1) throw exception
> > immediately
> > >> to fail fast and stop the world, 2) log the error and drop record and
> > >> proceed silently, 3) send such errors to a specific "error" Kafka
> topic,
> > >> or
> > >> record it as an app-level metrics (
> > >> https://kafka.apache.org/documentation/#kafka_streams_monitoring) for
> > >> monitoring.
> > >>
> > >> Guozhang
> > >>
> > >>
> > >>
> > >> On Fri, Oct 20, 2017 at 5:47 PM, Matt Farmer  wrote:
> > >>
> > >> > I did some more digging tonight.
> > >> >
> > >> > @Ted: It looks like the deserialization handler uses
> > >> > "default.deserialization.exception.handler" for the config name. No
> > >> > ".class" on the end. I'm inclined to think this should use
> > >> > "default.production.exception.handler".
> > >> >
> > >> > On Fri, Oct 20, 2017 at 8:22 PM Matt Farmer  wrote:
> > >> >
> > >> > > Okay, I've dug into this a little bit.
> > >> > >
> > >> > > I think getting access to the serialized record is possible, and
> > >> changing
> > >> > > the naming and return type is certainly doable. However, because
> > we're
> > >> > > hooking into the onCompletion callback we have no guarantee that
> the
> > >> > > ProcessorContext state hasn't changed by the time this particular
> > >> handler
> > >> > > runs. So I think the signature would 

[GitHub] kafka pull request #4149: KAFKA-6126: Reduce rebalance time by not checking ...

[jeyhunkarimov]

GitHub user jeyhunkarimov opened a pull request:

https://github.com/apache/kafka/pull/4149

KAFKA-6126: Reduce rebalance time by not checking if created topics are 
available



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeyhunkarimov/kafka KAFKA-6126

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4149.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4149


commit 94ba5ebae44b73b6a73fdb85ed6a9d527923c0d7
Author: Jeyhun Karimov 
Date:   2017-10-27T23:18:49Z

Reduce rebalance time by not checking if created topics are available




---

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Guozhang Wang]

> Please correct me if I'm wrong, but my understanding is that the record
> metadata is always null if an exception occurred while trying to produce.

That is right. Thanks.

I looked at the example code, and one thing I realized that since we are
not passing the context in the handle function, we may not be implement the
logic to send the fail records into another queue for future processing.
Would people think that would be a big issue?


Guozhang


On Thu, Oct 26, 2017 at 12:14 PM, Matt Farmer  wrote:

> Hello all,
>
> I've updated the KIP based on this conversation, and made it so that its
> interface, config setting, and parameters line up more closely with the
> interface in KIP-161 (deserialization handler).
>
> I believe there are a few specific questions I need to reply to.
>
> > The question I had about then handle parameters are around the record,
> > should it be `ProducerRecord`, or be generics of
> > `ProducerRecord` or `ProducerRecord > Object, ? extends Object>`?
>
> At this point in the code we're guaranteed that this is a
> ProducerRecord, so the generics would just make it harder
> to work with the key and value.
>
> > Also, should the handle function include the `RecordMetadata` as well in
> > case it is not null?
>
> Please correct me if I'm wrong, but my understanding is that the record
> metadata is always null if an exception occurred while trying to produce.
>
> > We may probably try to write down at least the following handling logic
> and
> > see if the given API is sufficient for it
>
> I've added some examples to the KIP. Let me know what you think.
>
> Cheers,
> Matt
>
> On Mon, Oct 23, 2017 at 9:00 PM Matt Farmer  wrote:
>
> > Thanks for this feedback. I’m at a conference right now and am planning
> on
> > updating the KIP again with details from this conversation later this
> week.
> >
> > I’ll shoot you a more detailed response then! :)
> > On Mon, Oct 23, 2017 at 8:16 PM Guozhang Wang 
> wrote:
> >
> >> Thanks for the KIP Matt.
> >>
> >> Regarding the handle interface of ProductionExceptionHandlerResponse,
> >> could
> >> you write it on the wiki also, along with the actual added config names
> >> (e.g. what
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+
> deserialization+exception+handlers
> >> described).
> >>
> >> The question I had about then handle parameters are around the record,
> >> should it be `ProducerRecord`, or be generics of
> >> `ProducerRecord` or `ProducerRecord >> Object, ? extends Object>`?
> >>
> >> Also, should the handle function include the `RecordMetadata` as well in
> >> case it is not null?
> >>
> >> We may probably try to write down at least the following handling logic
> >> and
> >> see if the given API is sufficient for it: 1) throw exception
> immediately
> >> to fail fast and stop the world, 2) log the error and drop record and
> >> proceed silently, 3) send such errors to a specific "error" Kafka topic,
> >> or
> >> record it as an app-level metrics (
> >> https://kafka.apache.org/documentation/#kafka_streams_monitoring) for
> >> monitoring.
> >>
> >> Guozhang
> >>
> >>
> >>
> >> On Fri, Oct 20, 2017 at 5:47 PM, Matt Farmer  wrote:
> >>
> >> > I did some more digging tonight.
> >> >
> >> > @Ted: It looks like the deserialization handler uses
> >> > "default.deserialization.exception.handler" for the config name. No
> >> > ".class" on the end. I'm inclined to think this should use
> >> > "default.production.exception.handler".
> >> >
> >> > On Fri, Oct 20, 2017 at 8:22 PM Matt Farmer  wrote:
> >> >
> >> > > Okay, I've dug into this a little bit.
> >> > >
> >> > > I think getting access to the serialized record is possible, and
> >> changing
> >> > > the naming and return type is certainly doable. However, because
> we're
> >> > > hooking into the onCompletion callback we have no guarantee that the
> >> > > ProcessorContext state hasn't changed by the time this particular
> >> handler
> >> > > runs. So I think the signature would change to something like:
> >> > >
> >> > > ProductionExceptionHandlerResponse handle(final ProducerRecord<..>
> >> > record,
> >> > > final Exception exception)
> >> > >
> >> > > Would this be acceptable?
> >> > >
> >> > > On Thu, Oct 19, 2017 at 7:33 PM Matt Farmer  wrote:
> >> > >
> >> > >> Ah good idea. Hmmm. I can line up the naming and return type but
> I’m
> >> not
> >> > >> sure if I can get my hands on the context and the record itself
> >> without
> >> > >> other changes.
> >> > >>
> >> > >> Let me dig in and follow up here tomorrow.
> >> > >> On Thu, Oct 19, 2017 at 7:14 PM Matthias J. Sax <
> >> matth...@confluent.io>
> >> > >> wrote:
> >> > >>
> >> > >>> Thanks for the KIP.
> >> > >>>
> >> > >>> Are you familiar with KIP-161?
> >> > >>>
> >> > >>>
> >> > >>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+

Re: Producer performance is awful when acks=all

[Colin McCabe]

Hi Vijay,

It is surprising to see such a large drop.  The first thing that it
suggests to me is that perhaps there is a network bottleneck that is
being glossed over in the acks=1 case, but not in the acks=all case.  In
the acks=1 case, is the broker co-located on the same machine as the
producer, or is it on a separate machine?

What kind of network are you using?  Is it gigabit ethernet?  10gigE?

If you run all three brokers on a single computer rather than on
multiple computes, do the results change?  If so, it suggests that the
network is acting as the bottleneck, again.

Are you using the new producer and consumer, or old versions?

best,
Colin


On Fri, Oct 27, 2017, at 14:42, Vijay Prakash wrote:
> Hey all,
> 
> I'm currently running Kafka 0.10.1 on Windows and am doing some perf
> testing. I tried out the perf test setup described in this blog post:
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines.
> When I try "single producer thread, 3x asynchronous replication", I get
> about 550k records/sec which seems acceptable for the perf loss due to
> running on Windows. However, when I set acks=all to try synchronous
> replication, I drop to about 120k records/sec, which is a LOT worse than
> the numbers in the blog post. Are there some specific settings that might
> cause acks=all to cause more perf issues? My broker settings are below.
> Thanks!
> 
> Broker settings:
> advertised.host.name = null
> advertised.listeners = null
> advertised.port = null
> authorizer.class.name =
> auto.create.topics.enable = false
> auto.leader.rebalance.enable = true
> background.threads = 10
> broker.id = 3
> broker.id.generation.enable = true
> broker.rack = sc3
> compression.type = producer
> connections.max.idle.ms = 60
> controlled.shutdown.enable = true
> controlled.shutdown.max.retries = 3
> controlled.shutdown.retry.backoff.ms = 5000
> controller.socket.timeout.ms = 3
> default.replication.factor = 1
> delete.topic.enable = true
> fetch.purgatory.purge.interval.requests = 1000
> group.max.session.timeout.ms = 30
> group.min.session.timeout.ms = 6000
> host.name =
> inter.broker.protocol.version = 0.10.1-IV2
> leader.imbalance.check.interval.seconds = 300
> leader.imbalance.per.broker.percentage = 10
> listeners = 
> log.cleaner.backoff.ms = 15000
> log.cleaner.dedupe.buffer.size = 134217728
> log.cleaner.delete.retention.ms = 8640
> log.cleaner.enable = true
> log.cleaner.io.buffer.load.factor = 0.9
> log.cleaner.io.buffer.size = 524288
> log.cleaner.io.max.bytes.per.second =
> 1.7976931348623157E308
> log.cleaner.min.cleanable.ratio = 0.5
> log.cleaner.min.compaction.lag.ms = 0
> log.cleaner.threads = 1
> log.cleanup.policy = [delete]
> log.dir =
> 
> D:/data/kafka/kafka-logs,H:/data/kafka/kafka-logs,I:/data/kafka/kafka-logs,J:/data/kafka/kafka-logs
> log.dirs = null
> log.flush.interval.messages = 2
> log.flush.interval.ms = 1000
> log.flush.offset.checkpoint.interval.ms = 6
> log.flush.scheduler.interval.ms = 9223372036854775807
> log.index.interval.bytes = 4096
> log.index.size.max.bytes = 10485760
> log.message.format.version = 0.10.1-IV2
> log.message.timestamp.difference.max.ms =
> 9223372036854775807
> log.message.timestamp.type = CreateTime
> log.preallocate = false
> log.retention.bytes = -1
> log.retention.check.interval.ms = 30
> log.retention.hours = 24
> log.retention.minutes = null
> log.retention.ms = null
> log.roll.hours = 168
> log.roll.jitter.hours = 0
> log.roll.jitter.ms = null
> log.roll.ms = null
> log.segment.bytes = 536870912
> log.segment.delete.delay.ms = 6
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> message.max.bytes = 112
> metric.reporters = []
> metrics.num.samples = 2
> metrics.sample.window.ms = 3
> min.insync.replicas = 3
> num.io.threads = 20

[DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

[Jun Rao]

Hi, Everyone,

We created "KIP-217: Expose a timeout to allow an expired ZK session to be
re-created".

https://cwiki.apache.org/confluence/display/KAFKA/KIP-217%3A+Expose+a+timeout+to+allow+an+expired+ZK+session+to+be+re-created

Please take a look and provide your feedback.

Thanks,

Jun

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

[Randall Hauch]

Yes, connector names should not be blank or contain just whitespace. In
fact, I might recommend that we trim whitespace at the front and rear of
new connector names and then disallowing any zero-length name. Existing
connectors would remain valid, and this would not break backward
compatibility. That might require a small kip simply to update the
documentation and specify what names are valid.

WDYT?

Randall

On Fri, Oct 27, 2017 at 1:08 PM, Colin McCabe  wrote:

> On Wed, Oct 25, 2017, at 01:07, Sönke Liebau wrote:
> > I've spent some time looking at this and testing various characters and
> > it
> > would appear that Randall's suspicion was spot on. I think we can support
> > a
> > fairly large set of characters with very minor changes.
> >
> > I was put of by the exceptions that were thrown when creating connectors
> > with certain characters and suspected a larger underlying problem when in
> > fact the only issue is, that the URL in the rest request used to retrieve
> > the response for the create connector request needs to be percent encoded
> > [1].
> >
> > I've fixed this and done some local testing which worked out quite
> > nicely,
> > apart from two special cases, I've not been able to find characters that
> > created issues, even space and slash work.
> > The mentioned special cases are:
> >   \  - if the name contains a backslash that is not the beginning of a
> > valid escape sequence the request fails before we ever get it in
> > ConnectorsResource, so a backslash would need to be escaped: \\
> >   "  - Quotation marks need to be escaped as well to keep the json body
> >   of
> > the request legal: \"
> > In both cases the escape character will be part of the connector name and
> > need to be specified in the url to retrieve the connector as well, even
> > though we could URL encode it in a legal way without escaping here. So
> > they
> > work, not sure if I'd recommend using those characters, but no real
> > reason
> > to prohibit people from using them that I can see either.
>
> Good research, Sönke.
>
> >
> >
> > What I'd do going forward is:
> > - withdraw the KIP, as I don't see a real need for one, since this is not
> > changing anything, just fixing things.
> > - add a section to the documentation around legal characters, specify the
> > ones I tested explicitly (url encoded %20 - %7F) and mention that most
> > other characters should work as well but no guarantees are given
> > - update the pull request for KAFKA-4930 to allow all characters but
> > still
> > prohibit creating a connector with an empty name. I'd propose to keep the
> > validator though as it'll give us a central location to do any checking
> > that might turn out to be necessary later on.
>
> Are empty names currently allowed?  That's unfortunate.
>
> > - add some integration tests to check connectors with special characters
> > in
> > their names work
> > - fix the url encoding line in ConnectorsResource
> >
> > Does that sound fair to everybody?
>
> It sounds good to me, but I will let someone more knowledgeable about
> connect chime in.
>
> best,
> Colin
>
> >
> > Kind regards,
> > Sönke
> >
> > [1]
> > https://github.com/apache/kafka/blob/trunk/connect/
> runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/
> ConnectorsResource.java#L102
> >
> > On Tue, Oct 24, 2017 at 8:40 PM, Colin McCabe 
> wrote:
> >
> > > On Tue, Oct 24, 2017, at 11:28, Sönke Liebau wrote:
> > > > Hi,
> > > >
> > > > after reading your messages I'll grant that I might have picked a
> > > > somewhat
> > > > draconic option to solve these issues.
> > > >
> > > > In general I believe that properly encoding the URLs after having
> created
> > > > the connectors should solve a lot of the issues already. For some
> > > > characters the rest api returns an error on creating the connector as
> > > > well,
> > > > so for that URL encoding won't help. However the connectors do get
> > > > created
> > > > even though an error is returned, I've never investigated if they
> are in
> > > > a
> > > > consistent state tbh - I'll give this another look.
> > > >
> > > > @colin: Entity encoding would allow us to encode a lot of characters,
> > > > however I am unsure whether we should prefer it over url encoding in
> this
> > > > case, as mostly the end user would have to encode the characters
> himself.
> > > > And due to entity encoding ending every character with a ; which
> causes
> > > > the
> > > > embedded jetty server to cut the connector name at that character
> we'd
> > > > probably need to encode that character in URL encoding again for
> that to
> > > > work out - which might get a bit too complex tbh.
> > >
> > > Sorry, I meant to write percent-encoding, not entity refs.
> > > https://en.wikipedia.org/wiki/Percent-encoding
> > >
> > > best,
> > > Colin
> > >
> > >
> > > > I will further investigate which characters the url decoding that
> jetty
> > > > brings to the table will let us use 

Producer performance is awful when acks=all

[Vijay Prakash]

Hey all,

I'm currently running Kafka 0.10.1 on Windows and am doing some perf testing. I 
tried out the perf test setup described in this blog post: 
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines.
 When I try "single producer thread, 3x asynchronous replication", I get about 
550k records/sec which seems acceptable for the perf loss due to running on 
Windows. However, when I set acks=all to try synchronous replication, I drop to 
about 120k records/sec, which is a LOT worse than the numbers in the blog post. 
Are there some specific settings that might cause acks=all to cause more perf 
issues? My broker settings are below. Thanks!

Broker settings:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
authorizer.class.name =
auto.create.topics.enable = false
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 3
broker.id.generation.enable = true
broker.rack = sc3
compression.type = producer
connections.max.idle.ms = 60
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 3
default.replication.factor = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.max.session.timeout.ms = 30
group.min.session.timeout.ms = 6000
host.name =
inter.broker.protocol.version = 0.10.1-IV2
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listeners = 
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 8640
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = 
D:/data/kafka/kafka-logs,H:/data/kafka/kafka-logs,I:/data/kafka/kafka-logs,J:/data/kafka/kafka-logs
log.dirs = null
log.flush.interval.messages = 2
log.flush.interval.ms = 1000
log.flush.offset.checkpoint.interval.ms = 6
log.flush.scheduler.interval.ms = 9223372036854775807
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 0.10.1-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 30
log.retention.hours = 24
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 536870912
log.segment.delete.delay.ms = 6
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 112
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 3
min.insync.replicas = 3
num.io.threads = 20
num.network.threads = 20
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 60
offsets.retention.minutes = 1440
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 5
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = class 
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
producer.purgatory.purge.interval.requests = 1000

Call for Paper: Kafka Summit London 2018

[Gwen Shapira]

Hi Kafka Users and Contributors,

We are having the first ever Kafka Summit that is really close to Europe
even if not quite part of Europe.

Call for papers is now open and you should all submit abstracts and tell
your colleagues to submit abstracts too! Seriously, if you use Kafka,
whether beginner or advanced, I'm sure you have stories to share and we
really want to hear all about it.

Every year some people ask me "I have a pretty small cluster, are you still
interested?". Yes! innovative use-cases are as important than huge scale.
Probably more so. We are looking for diverse agenda - different industries,
experiences, stream processing technologies, programming languages, data
stores, job descriptions, use-cases, sizes, colors, genders, ages, etc,
etc. We want the conference to have a cool talk for everyone.

Here's the important part:

Submit your proposal here
:
*https://kafka-summit.org/kafka-summit-london-2018/speakers/
*

Important dates to note:

   -

   Call for Papers closes: December 1, 2017
   -

   Notifications sent: December 20, 2017
   -

   Presentations due for initial review: March 19, 2018
   -

   Presentations due for final approval: April 9, 2018


I hope to see you in London! Registration will open soon!


Gwen Shapira Kafka PMC and conference enthusiast

[GitHub] kafka pull request #4148: KAFKA-6120: RecordCollector should not retry sendi...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4148

KAFKA-6120: RecordCollector should not retry sending



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-6120-recordCollector

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4148.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4148


commit f99a991dce619bd314af244b1ebabcbc7f6e1f9f
Author: Matthias J. Sax 
Date:   2017-10-27T19:22:03Z

KAFKA-6120: RecordCollector should not retry sending




---

[GitHub] kafka pull request #4147: MINOR: Fix inconsistency in StopReplica/LeaderAndI...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/4147

MINOR: Fix inconsistency in StopReplica/LeaderAndIsr error counts



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka fix-error-inconsistencies

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4147.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4147


commit a3cc429885dc4b998ec4933f2e16ac1c14659d36
Author: Jason Gustafson 
Date:   2017-10-27T19:05:25Z

MINOR: Fix inconsistency in StopReplica/LeaderAndIsr error counts




---

Build failed in Jenkins: kafka-0.11.0-jdk7 #330

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6119: Bump epoch when expiring transactions in the

--
[...truncated 7.52 KB...]
if (!Console.readLine().equalsIgnoreCase("y")) {
 ^
:95:
 object FetchRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
FetchRequest.batchByTopic(data)
^
:49:
 object FetchRequest in package api is deprecated: This object has been 
deprecated and will be removed in a future release.
  val underlying = kafka.api.FetchRequest(
 ^
:67:
 comparing values of types Long and Null using `==' will always yield false
if(l == null)
 ^
:202:
 method round in class RichLong is deprecated: This is an integer type; there 
is no reason to round it.  Perhaps you meant to call this on a floating-point 
value?
throttleTimeMs = throttleTime(clientMetric, 
getQuotaMetricConfig(clientQuotaEntity.quota)).round.toInt

   ^
:308:
 class ZKGroupTopicDirs in package utils is deprecated: This class has been 
deprecated and will be removed in a future release.
val topicDirs = new ZKGroupTopicDirs(offsetCommitRequest.groupId, 
topicPartition.topic)
^
:347:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
  if (partitionData.timestamp == 
OffsetCommitRequest.DEFAULT_TIMESTAMP)
^
:350:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
offsetRetention + partitionData.timestamp
^
:650:
 method offsetData in class ListOffsetRequest is deprecated: see corresponding 
Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {
 ^
:650:
 class PartitionData in object ListOffsetRequest is deprecated: see 
corresponding Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {

  ^
:655:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  new ListOffsetResponse.PartitionData(Errors.UNKNOWN_TOPIC_OR_PARTITION, 
List[JLong]().asJava)
  ^
:680:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
(topicPartition, new ListOffsetResponse.PartitionData(Errors.NONE, 
offsets.map(new JLong(_)).asJava))
 ^
:687:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e), List[JLong]().asJava))
   ^
:690:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e), List[JLong]().asJava))
   ^
:1016:
 class ZKGroupTopicDirs in 

[GitHub] kafka-site pull request #104: Replace link

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/104


---

[GitHub] kafka-site issue #104: Replace link

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/104
  
Merged to asf-site.


---

[GitHub] kafka-site issue #104: Replace link

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/104
  
LGTM. Could you submit a PR for changes in ` 0110/streams` as well in 
`kafka` repo?


---

Build failed in Jenkins: kafka-trunk-jdk8 #2177

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Ensure that the producer in testAlterReplicaLogDirs is always

[ismael] MINOR: Remove TLS renegotiation code

--
[...truncated 1.88 MB...]

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails PASSED

Re: [VOTE] KIP-207:The Offsets which ListOffsetsResponse returns should monotonically increase even during a partition leader change

[Colin McCabe]

Thanks, everyone.  I'm going to close the vote tomorrow if there are no
more comments or votes.

regards,
Colin


On Thu, Oct 26, 2017, at 08:09, Manikumar wrote:
> Thanks for the KIP.
> +1 (non-binding)
> 
> 
> On Thu, Oct 26, 2017 at 5:58 AM, Jason Gustafson 
> wrote:
> 
> > +1. Thanks for the KIP.
> >
> > On Mon, Oct 23, 2017 at 11:30 AM, Colin McCabe  wrote:
> >
> > > On Mon, Oct 23, 2017, at 10:29, Jason Gustafson wrote:
> > > > Thanks for the KIP. I'm assuming the new behavior only affects
> > > > ListOffsets requests from the consumer.
> > >
> > > That's a very good point.  I will add a caveat that we only apply the
> > > KIP-207 behavior to requests from clients, not requests from other
> > > brokers (such as the ones made by ReplicaFetcherThread).
> > >
> > > > Might be worth mentioning that in the KIP.
> > > > Also, does it affect all ListOffsets requests, or only those that
> > specify
> > > > the latest offset?
> > >
> > > I don't feel great about allowing someone to ask for the offset at time
> > > T, get back X, and then ask again for the offset at T the next second
> > > and get back InvalidOffsetException.  So it's probably best just to
> > > apply the KIP-207 behavior to all ListOffsets requests from consumers.
> > >
> > > Thinking about it a bit more, we should disable the KIP-207 behavior
> > > when unclean leader elections are enabled on the broker.  When unclean
> > > leader elections are enabled, data loss is possible.  So we cannot
> > > guarantee that offsets will always go forwards, even in theory, in this
> > > mode.
> > >
> > > I update the kip-- check it out.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Oct 18, 2017 at 9:15 AM, Colin McCabe 
> > > wrote:
> > > >
> > > > > On Wed, Oct 18, 2017, at 04:09, Ismael Juma wrote:
> > > > > > Thanks for the KIP, +1 (binding). A few comments:
> > > > > >
> > > > > > 1. I agree with Jun about LEADER_NOT_AVAILABLE for the error code
> > for
> > > > > > older
> > > > > > versions.
> > > > > > 2. OffsetNotAvailableException seems clear enough (i.e. we don't
> > > need the
> > > > > > "ForPartition" part)
> > > > >
> > > > > Yeah, that is shorter and probably clearer.  Changed.
> > > > >
> > > > > > 3. The KIP seems to be missing the compatibility section.
> > > > >
> > > > > Added.
> > > > >
> > > > > > 4. It would be good to mention that it's now possible for a fetch
> > to
> > > > > > succeed while list offsets will not for a period of time. And for
> > > older
> > > > > > versions, the latter will return LeaderNotAvailable while the
> > former
> > > > > > would
> > > > > > work fine, which is a bit unexpected. Not much we can do about it,
> > > but
> > > > > > worth mentioning it in my opinion.
> > > > >
> > > > > Fair enough
> > > > >
> > > > > cheers,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Oct 17, 2017 at 9:26 PM, Jun Rao  wrote:
> > > > > >
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the KIP. +1. Just a minor comment. For the old client
> > > > > requests,
> > > > > > > would it be better to return a LEADER_NOT_AVAILABLE error
> > instead?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Oct 17, 2017 at 11:11 AM, Colin McCabe <
> > cmcc...@apache.org
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to start the voting process for KIP-207:The  Offsets
> > > which
> > > > > > > > ListOffsetsResponse returns should monotonically increase even
> > > > > during a
> > > > > > > > partition leader change.
> > > > > > > >
> > > > > > > > See
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > 207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+
> > > > > > > > monotonically+increasing+even+during+a+partition+leader+change
> > > > > > > > for details.
> > > > > > > >
> > > > > > > > The voting process will run for at least 72 hours.
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > Colin
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

[Colin McCabe]

On Wed, Oct 25, 2017, at 01:07, Sönke Liebau wrote:
> I've spent some time looking at this and testing various characters and
> it
> would appear that Randall's suspicion was spot on. I think we can support
> a
> fairly large set of characters with very minor changes.
> 
> I was put of by the exceptions that were thrown when creating connectors
> with certain characters and suspected a larger underlying problem when in
> fact the only issue is, that the URL in the rest request used to retrieve
> the response for the create connector request needs to be percent encoded
> [1].
> 
> I've fixed this and done some local testing which worked out quite
> nicely,
> apart from two special cases, I've not been able to find characters that
> created issues, even space and slash work.
> The mentioned special cases are:
>   \  - if the name contains a backslash that is not the beginning of a
> valid escape sequence the request fails before we ever get it in
> ConnectorsResource, so a backslash would need to be escaped: \\
>   "  - Quotation marks need to be escaped as well to keep the json body
>   of
> the request legal: \"
> In both cases the escape character will be part of the connector name and
> need to be specified in the url to retrieve the connector as well, even
> though we could URL encode it in a legal way without escaping here. So
> they
> work, not sure if I'd recommend using those characters, but no real
> reason
> to prohibit people from using them that I can see either.

Good research, Sönke.

> 
> 
> What I'd do going forward is:
> - withdraw the KIP, as I don't see a real need for one, since this is not
> changing anything, just fixing things.
> - add a section to the documentation around legal characters, specify the
> ones I tested explicitly (url encoded %20 - %7F) and mention that most
> other characters should work as well but no guarantees are given
> - update the pull request for KAFKA-4930 to allow all characters but
> still
> prohibit creating a connector with an empty name. I'd propose to keep the
> validator though as it'll give us a central location to do any checking
> that might turn out to be necessary later on.

Are empty names currently allowed?  That's unfortunate.

> - add some integration tests to check connectors with special characters
> in
> their names work
> - fix the url encoding line in ConnectorsResource
> 
> Does that sound fair to everybody?

It sounds good to me, but I will let someone more knowledgeable about
connect chime in.

best,
Colin

> 
> Kind regards,
> Sönke
> 
> [1]
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java#L102
> 
> On Tue, Oct 24, 2017 at 8:40 PM, Colin McCabe  wrote:
> 
> > On Tue, Oct 24, 2017, at 11:28, Sönke Liebau wrote:
> > > Hi,
> > >
> > > after reading your messages I'll grant that I might have picked a
> > > somewhat
> > > draconic option to solve these issues.
> > >
> > > In general I believe that properly encoding the URLs after having created
> > > the connectors should solve a lot of the issues already. For some
> > > characters the rest api returns an error on creating the connector as
> > > well,
> > > so for that URL encoding won't help. However the connectors do get
> > > created
> > > even though an error is returned, I've never investigated if they are in
> > > a
> > > consistent state tbh - I'll give this another look.
> > >
> > > @colin: Entity encoding would allow us to encode a lot of characters,
> > > however I am unsure whether we should prefer it over url encoding in this
> > > case, as mostly the end user would have to encode the characters himself.
> > > And due to entity encoding ending every character with a ; which causes
> > > the
> > > embedded jetty server to cut the connector name at that character we'd
> > > probably need to encode that character in URL encoding again for that to
> > > work out - which might get a bit too complex tbh.
> >
> > Sorry, I meant to write percent-encoding, not entity refs.
> > https://en.wikipedia.org/wiki/Percent-encoding
> >
> > best,
> > Colin
> >
> >
> > > I will further investigate which characters the url decoding that jetty
> > > brings to the table will let us use and if all of these are correctly
> > > handled during connector creation and report back with a new list of
> > > characters that I think we can support fairly easily.
> > >
> > > Kind regards,
> > > Sönke
> > >
> > >
> > > On Tue, Oct 24, 2017 at 6:42 PM, Colin McCabe 
> > wrote:
> > >
> > > > It should be possible to use entity references to encode these
> > > > characters in URLs.  See https://dev.w3.org/html5/html-author/charref
> > > > Maybe I'm misunderstanding the problem, but can we simply encode the
> > > > URLs, rather than restricting the names?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Oct 23, 2017, at 14:12, Randall Hauch wrote:
> > > > > 

Re: [DISCUSS] KIP-209 Connection String Support

[Colin McCabe]

On Tue, Oct 24, 2017, at 22:51, Michael André Pearce wrote:
> Fair enough on URL encoding but as mentioned it is important to be able
> to escape, I agree with backslash option.
> 
> I would still like some form of prefix to the string to denote it is for
> kafka.

I don't think a prefix is necessary here.  URLs have a prefix because
there are multiple services which you could access (HTTP vs HTTPS, etc.)
 If you are passing a string to Kafka, the string is for Kafka, not for
something else, so the issue doesn't exist.

best,
Colin


> 
>  E.g. kafka:: (if semicolon separators)
> 
> Sent from my iPad
> 
> > On 24 Oct 2017, at 17:27, Colin McCabe  wrote:
> > 
> > Hi Clebert,
> > 
> > As some other people mentioned, a comma is probably not a great choice
> > for the entry separator.  We have a lot of configuration values that
> > already include commas.  How about using a semicolon instead?
> > 
> > You also need an escaping system in case someone needs a semicolon (or
> > whatever) that is part of a configuration key or configuration value. 
> > How about a simple backslash?  And then if you want a literal backslash,
> > you put in two backslashes.
> > 
> >> On Thu, Oct 19, 2017, at 18:10, Michael André Pearce wrote:
> >> Just another point to why I’d propose the below change to the string
> >> format I propose , is an ability to encode the strings easily.
> >> 
> >> We should note that it’s quite typical for serializers to user a
> >> schematic registry where one of their properties they will need to set
> >> would be in some form like:
> >> 
> >> schema.registry.url=http://schema1:80,schema2:80/api
> >> 
> >> So being able to safely encode this is important. 
> >> 
> >> Sent from my iPhone
> >> 
> >>> On 20 Oct 2017, at 01:47, Michael André Pearce 
> >>>  wrote:
> >>> 
> >>> Hi Clebert
> >>> 
> >>> Great kip!
> >>> 
> >>> Instead of ‘;’ to separate the host sections with the params section 
> >>> could it be a ‘?’
> >>> 
> >>> And like wise ‘,’ param separator could this be ‘&’ (keep the ‘,’ for 
> >>> host separator just makes easier to distinguish)
> >>> 
> >>> Also this was it makes it easier to encode params etc as can just re use 
> >>> url encoders.
> > 
> > Please, no.  URL encoders will mangle a lot of things horribly (like
> > less than signs, greater than signs, etc.)  We should not make this a
> > URL or pseudo-URL (see the discussion above).  We should make it clear
> > that this is not a URL.
> > 
> >> Invalid conversions would throw InvalidArgumentException (with a 
> >> description of the invalid conversion)
> >> Invalid parameters would throw InvalidArgumentException (with the name of 
> >> the invalid parameter).
> > 
> > This will cause a lot of compatibility problems, right?  If I switch
> > back and forth between two Kafka versions, they will support slightly
> > different sets of configuration parameters.  It seems saner to simply
> > ignore configuration parameters that we don't understand, like we do
> > now.
> > 
> > best,
> > Colin
> > 
> > 
> >>> 
> >>> Also as like many systems it typical to note what the connection string 
> >>> is for with a prefix eg ‘kafka://‘
> >>> 
> >>> Just makes it obvious when an app has a list of connection strings in 
> >>> their runtime properties which is for which technology.
> >>> 
> >>> Eg example connection string would be:
> >>> 
> >>> kafka://host1:port1,host2:port2?param1=value1=value2
> >>> 
> >>> Cheers
> >>> Mike
> >>> 
> >>> Sent from my iPhone
> >>> 
>  On 19 Oct 2017, at 19:29, Clebert Suconic  
>  wrote:
>  
>  Do I have to do anything here?
>  
>  I wonder how long I need to wait before proposing the vote.
>  
>  On Tue, Oct 17, 2017 at 1:17 PM, Clebert Suconic
>   wrote:
> > I had these updates in already... you just changed the names at the
> > string.. but it was pretty much the same thing I think... I had taken
> > you suggestions though.
> > 
> > 
> > The Exceptions.. these would be implementation details... all I wanted
> > to make sure is that users would get the name of the invalid parameter
> > as part of a string on a message.
> > 
> > On Tue, Oct 17, 2017 at 3:15 AM, Satish Duggana
> >  wrote:
> >> You may need to update KIP with the details discussed in this thread in
> >> proposed changes section.
> >> 
>  My proposed format for the connection string would be:
>  IP1:host1,IP2:host2,...IPN:hostn;parameterName=value1;parameterName2=value2;...
> >> parameterNameN=valueN
> >> Format should be
> >> host1:port1,host2:port2,…host:portn;param-name1=param-val1,..
> >> 
>  Invalid conversions would throw InvalidArgumentException (with a
> >> description of the invalid conversion)
>  Invalid parameters would throw InvalidArgumentException (with the 
>  

[VOTE] 1.0.0 RC4

[Guozhang Wang]

Hello Kafka users, developers and client-developers,

This is the fifth candidate for release of Apache Kafka 1.0.0. The main PRs
that gets merged in after RC3 are the following:

*https://github.com/apache/kafka/commit/def1a768a6301c14ad6611358716ab03de04e76b
*

*https://github.com/apache/kafka/commit/b9fc0f2e6892062efa1fff0c6f7bfc683c8ba7ab
*

*https://github.com/apache/kafka/commit/a51fdcd2ee7efbd14857448a2fb7ecb71531e1f9
*

*https://github.com/apache/kafka/commit/109a60c77a56d4afed488c3ba35dc8459fde15ce
*

It's worth noting that starting in this version we are using a different
version protocol with three digits: *major.minor.bug-fix*

Any and all testing is welcome, but the following areas are worth
highlighting:

1. Client developers should verify that their clients can produce/consume
to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
2. Performance and stress testing. Heroku and LinkedIn have helped with
this in the past (and issues have been found and fixed).
3. End users can verify that their apps work correctly with the new release.

This is a major version release of Apache Kafka. It includes 29 new KIPs.
See the release notes and release plan
(*https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
*)
for more details. A few feature highlights:

* Java 9 support with significantly faster TLS and CRC32C implementations
* JBOD improvements: disk failure only disables failed disk but not the
broker (KIP-112/KIP-113 part I)
* Controller improvements: reduced logging change to greatly accelerate
admin request handling.
* Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
KIP-188, KIP-196)
* Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
and drop compatibility "Evolving" annotations

Release notes for the 1.0.0 release:
*http://home.apache.org/~guozhang/kafka-1.0.0-rc4/RELEASE_NOTES.html
*



*** Please download, test and vote by Tuesday, October 31, 8pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
*http://home.apache.org/~guozhang/kafka-1.0.0-rc4/
*

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
*http://home.apache.org/~guozhang/kafka-1.0.0-rc4/javadoc/
*

* Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc4 tag:

https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d4a3919e408e444dde5db5a261c6f912cb8475a2

* Documentation:
Note the documentation can't be pushed live due to changes that will not go
live until the release. You can manually verify by downloading
http://home.apache.org/~guozhang/kafka-1.0.0-rc4/kafka_2.11-1.0.0-site-docs.tgz


The Jenkins builders for this RC can now be found here:

System test (still running):
*https://jenkins.confluent.io/job/system-test-kafka-1.0/18/
*
Unit test: *https://builds.apache.org/job/kafka-1.0-jdk7/61/
*


/**


Thanks,
-- Guozhang

Build failed in Jenkins: kafka-1.0-jdk7 #64

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Mention "per listener" security overrides in

--
[...truncated 1.82 MB...]
org.apache.kafka.streams.KafkaStreamsTest > testStateGlobalThreadClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldThrowExceptionSettingUncaughtExceptionHandlerNotInCreateState STARTED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldThrowExceptionSettingUncaughtExceptionHandlerNotInCreateState PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldThrowExceptionSettingStateListenerNotInCreateState STARTED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldThrowExceptionSettingStateListenerNotInCreateState PASSED

org.apache.kafka.streams.KafkaStreamsTest > testNumberDefaultMetrics STARTED

org.apache.kafka.streams.KafkaStreamsTest > testNumberDefaultMetrics PASSED

org.apache.kafka.streams.KafkaStreamsTest > shouldReturnThreadMetadata STARTED

org.apache.kafka.streams.KafkaStreamsTest > shouldReturnThreadMetadata PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled 
STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfNotAtLestOnceOrExactlyOnce 

Re: [VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Ted Yu]

+1

On Fri, Oct 27, 2017 at 10:15 AM, Onur Karaman  wrote:

> I'd like to start the vote for KIP-214: Add
> zookeeper.max.in.flight.requests config to the broker
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
>
> - Onur
>

[VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Onur Karaman]

I'd like to start the vote for KIP-214: Add
zookeeper.max.in.flight.requests config to the broker

https://cwiki.apache.org/confluence/display/KAFKA/KIP-214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker

- Onur

Jenkins build is back to normal : kafka-trunk-jdk8 #2176

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #4146: MINOR: Tighten up locking when aborting expired tr...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/4146

MINOR: Tighten up locking when aborting expired transactions

This is a followup to #4137 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
MINOR-followups-to-bump-epoch-on-expire-patch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4146


commit 1c8cc6672f93315f1027fe7cf1dae9975cf7871e
Author: Apurva Mehta 
Date:   2017-10-27T16:55:33Z

Tighten up locking and address other minor comments




---

Jenkins build is back to normal : kafka-trunk-jdk7 #2930

[Apache Jenkins Server]

See 

Re: Errors when running integration/kafka/tools/MirrorMakerIntegrationTest

[Pavel Drankov]

Yes, I m interested and would like to make a PR.

Best wishes,
Pavel

On 27 October 2017 at 19:05, Ismael Juma  wrote:

> Yes, I think it would be good to eliminate these lines from the output. A
> PR is welcome if you're interested.
>
> Ismael
>
> On Fri, Oct 27, 2017 at 5:03 PM, Pavel Drankov 
> wrote:
>
> > Of course it don't fail tests. I mean to fix log appearance.
> >
> > Best wishes,
> > Pavel
> >
> > On 27 October 2017 at 18:48, Ted Yu  wrote:
> >
> > > The error log appears in other tests as well.
> > >
> > > I ran MirrorMakerIntegrationTest on trunk just now - it passed.
> > > I didn't see the test failing in any recent Jenkins builds.
> > >
> > > There're flaky tests, such as KAFKA-6137
> > >
> > > FYI
> > >
> > >
> > > On Fri, Oct 27, 2017 at 8:35 AM, Pavel Drankov 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I see some error logs when running
> > > > integration/kafka/tools/MirrorMakerIntegrationTest.
> > > >
> > > > Log:
> > > >
> > > > [2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered,
> so
> > > > ZooKeeper server won't take any action on ERROR or SHUTDOWN server
> > state
> > > > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> > > > [2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered,
> so
> > > > ZooKeeper server won't take any action on ERROR or SHUTDOWN server
> > state
> > > > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> > > >
> > > > Is it make sense to fix it providing a ShutdownHandler?
> > > >
> > > > Best wishes,
> > > > Pavel
> > > >
> > >
> >
>

Re: Errors when running integration/kafka/tools/MirrorMakerIntegrationTest

[Ismael Juma]

Yes, I think it would be good to eliminate these lines from the output. A
PR is welcome if you're interested.

Ismael

On Fri, Oct 27, 2017 at 5:03 PM, Pavel Drankov  wrote:

> Of course it don't fail tests. I mean to fix log appearance.
>
> Best wishes,
> Pavel
>
> On 27 October 2017 at 18:48, Ted Yu  wrote:
>
> > The error log appears in other tests as well.
> >
> > I ran MirrorMakerIntegrationTest on trunk just now - it passed.
> > I didn't see the test failing in any recent Jenkins builds.
> >
> > There're flaky tests, such as KAFKA-6137
> >
> > FYI
> >
> >
> > On Fri, Oct 27, 2017 at 8:35 AM, Pavel Drankov 
> > wrote:
> >
> > > Hi,
> > >
> > > I see some error logs when running
> > > integration/kafka/tools/MirrorMakerIntegrationTest.
> > >
> > > Log:
> > >
> > > [2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered, so
> > > ZooKeeper server won't take any action on ERROR or SHUTDOWN server
> state
> > > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> > > [2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered, so
> > > ZooKeeper server won't take any action on ERROR or SHUTDOWN server
> state
> > > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> > >
> > > Is it make sense to fix it providing a ShutdownHandler?
> > >
> > > Best wishes,
> > > Pavel
> > >
> >
>

Re: Errors when running integration/kafka/tools/MirrorMakerIntegrationTest

[Pavel Drankov]

Of course it don't fail tests. I mean to fix log appearance.

Best wishes,
Pavel

On 27 October 2017 at 18:48, Ted Yu  wrote:

> The error log appears in other tests as well.
>
> I ran MirrorMakerIntegrationTest on trunk just now - it passed.
> I didn't see the test failing in any recent Jenkins builds.
>
> There're flaky tests, such as KAFKA-6137
>
> FYI
>
>
> On Fri, Oct 27, 2017 at 8:35 AM, Pavel Drankov 
> wrote:
>
> > Hi,
> >
> > I see some error logs when running
> > integration/kafka/tools/MirrorMakerIntegrationTest.
> >
> > Log:
> >
> > [2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered, so
> > ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
> > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> > [2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered, so
> > ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
> > changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> >
> > Is it make sense to fix it providing a ShutdownHandler?
> >
> > Best wishes,
> > Pavel
> >
>

Re: Errors when running integration/kafka/tools/MirrorMakerIntegrationTest

[Ted Yu]

The error log appears in other tests as well.

I ran MirrorMakerIntegrationTest on trunk just now - it passed.
I didn't see the test failing in any recent Jenkins builds.

There're flaky tests, such as KAFKA-6137

FYI


On Fri, Oct 27, 2017 at 8:35 AM, Pavel Drankov  wrote:

> Hi,
>
> I see some error logs when running
> integration/kafka/tools/MirrorMakerIntegrationTest.
>
> Log:
>
> [2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered, so
> ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
> changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> [2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered, so
> ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
> changes (org.apache.zookeeper.server.ZooKeeperServer:472)
>
> Is it make sense to fix it providing a ShutdownHandler?
>
> Best wishes,
> Pavel
>

[GitHub] kafka pull request #4034: MINOR: Remove TLS renegotiation code

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4034


---

[jira] [Created] (KAFKA-6138) Simplify StreamsBuilder#addGlobalStore

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6138:
--

 Summary: Simplify StreamsBuilder#addGlobalStore
 Key: KAFKA-6138
 URL: https://issues.apache.org/jira/browse/KAFKA-6138
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax


{{StreamsBuilder#addGlobalStore}} is a 1:1 copy of {{Topology#addGlobalStore}}. 
However, at DSL level we can actually auto-generate the two processor names and 
thus simplify the interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk9 #157

[Apache Jenkins Server]

See 

Errors when running integration/kafka/tools/MirrorMakerIntegrationTest

[Pavel Drankov]

Hi,

I see some error logs when running
integration/kafka/tools/MirrorMakerIntegrationTest.

Log:

[2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered, so
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
[2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered, so
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state
changes (org.apache.zookeeper.server.ZooKeeperServer:472)

Is it make sense to fix it providing a ShutdownHandler?

Best wishes,
Pavel

Jenkins build is back to normal : kafka-1.0-jdk7 #63

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6109) ResetIntegrationTest may fail due to IllegalArgumentException

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved KAFKA-6109.
---
Resolution: Cannot Reproduce

> ResetIntegrationTest may fail due to IllegalArgumentException
> -
>
> Key: KAFKA-6109
> URL: https://issues.apache.org/jira/browse/KAFKA-6109
> Project: Kafka
>  Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> From https://builds.apache.org/job/kafka-trunk-jdk7/2918 :
> {code}
> org.apache.kafka.streams.integration.ResetIntegrationTest > 
> testReprocessingFromScratchAfterResetWithIntermediateUserTopic FAILED
> java.lang.IllegalArgumentException: Setting the time to 1508791687000 
> while current time 1508791687475 is newer; this is not allowed
> at 
> org.apache.kafka.common.utils.MockTime.setCurrentTimeMs(MockTime.java:81)
> at 
> org.apache.kafka.streams.integration.AbstractResetIntegrationTest.beforePrepareTest(AbstractResetIntegrationTest.java:114)
> at 
> org.apache.kafka.streams.integration.ResetIntegrationTest.before(ResetIntegrationTest.java:55)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6137) RestoreIntegrationTest sometimes fails with assertion error

[Ted Yu (JIRA)]

Ted Yu created KAFKA-6137:
-

 Summary: RestoreIntegrationTest sometimes fails with assertion 
error
 Key: KAFKA-6137
 URL: https://issues.apache.org/jira/browse/KAFKA-6137
 Project: Kafka
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


>From https://builds.apache.org/job/kafka-1.0-jdk7/62 :
{code}
org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldSuccessfullyStartWhenLoggingDisabled FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldSuccessfullyStartWhenLoggingDisabled(RestoreIntegrationTest.java:195)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Matthias J. Sax]

Thanks. I understand what you are saying, but I don't agree that

> but also we need a commit() method

I would just not provide `commit()` at DSL level and close the
corresponding Jira as "not a problem" or similar.


-Matthias

On 10/27/17 3:42 PM, Jeyhun Karimov wrote:
> Hi Matthias,
> 
> Thanks for your comments. I agree that this is not the best way to do. A
> bit of history behind this design.
> 
> Prior doing this, I tried to provide ProcessorContext itself as an argument
> in Rich interfaces. However, we dont want to give users that flexibility
> and “power”. Moreover, ProcessorContext contains processor level
> information and not Record level info. The only thing we need ij
> ProcessorContext is commit() method.
> 
> So, as far as I understood, we need recor context (offset, timestamp and
> etc) but also we need a commit() method ( we dont want to provide
> ProcessorContext as a parameter so users can use ProcessorContext.commit()
> ).
> 
> As a result, I thought to “propagate” commit() call from RecordContext to
> ProcessorContext() .
> 
> 
> If there is a misunderstanding in motvation/discussion of KIP/included
> jiras please let me know.
> 
> 
> Cheers,
> Jeyhun
> 
> 
> On Fri 27. Oct 2017 at 12:39, Matthias J. Sax  wrote:
> 
>> I am personally still not convinced, that we should add `commit()` at all.
>>
>> @Guozhang: you created the original Jira. Can you elaborate a little
>> bit? Isn't requesting commits a low level API that should not be exposed
>> in the DSL? Just want to understand the motivation better. Why would
>> anybody that uses the DSL ever want to request a commit? To me,
>> requesting commits is useful if you manipulated state explicitly, ie,
>> via Processor API.
>>
>> Also, for the solution: it seem rather unnatural to me, that we add
>> `commit()` to `RecordContext` -- from my understanding, `RecordContext`
>> is an helper object that provide access to record meta data. Requesting
>> a commit is something quite different. Additionally, a commit does not
>> commit a specific record but a `RecrodContext` is for a specific record.
>>
>> To me, this does not seem to be a sound API design if we follow this path.
>>
>>
>> -Matthias
>>
>>
>>
>> On 10/26/17 10:41 PM, Jeyhun Karimov wrote:
>>> Hi,
>>>
>>> Thanks for your suggestions.
>>>
>>> I have some comments, to make sure that there is no misunderstanding.
>>>
>>>
>>> 1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
 user to consolidate this call as
 "processorContext.recordContext().commit()". And internal implementation
 of
 `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
 this call.
>>>
>>>
>>> - I think we should not deprecate `ProcessorContext.commit()`. The main
>>> intuition that we introduce `commit()` in `RecordContext` is that,
>>> `RecordContext` is the one which is provided in Rich interfaces. So if
>> user
>>> wants to commit, then there should be some method inside `RecordContext`
>> to
>>> do so. Internally, `RecordContext.commit()` calls
>>> `ProcessorContext.commit()`  (see the last code snippet in KIP-159):
>>>
>>> @Override
>>> public void process(final K1 key, final V1 value) {
>>>
>>> recordContext = new RecordContext() {   //
>>> recordContext initialization is added in this KIP
>>> @Override
>>> public void commit() {
>>> context().commit();
>>> }
>>>
>>> @Override
>>> public long offset() {
>>> return context().recordContext().offset();
>>> }
>>>
>>> @Override
>>> public long timestamp() {
>>> return context().recordContext().timestamp();
>>> }
>>>
>>> @Override
>>> public String topic() {
>>> return context().recordContext().topic();
>>> }
>>>
>>> @Override
>>> public int partition() {
>>> return context().recordContext().partition();
>>> }
>>>   };
>>>
>>>
>>> So, we cannot deprecate `ProcessorContext.commit()` in this case IMO.
>>>
>>>
>>> 2. Add the `task` reference to the impl class, `ProcessorRecordContext`,
>> so
 that it can implement the commit call itself.
>>>
>>>
>>> - Actually, I don't think that we need `commit()` in
>>> `ProcessorRecordContext`. The main intuition is to "transfer"
>>> `ProcessorContext.commit()` call to Rich interfaces, to support
>>> user-specific committing.
>>>  To do so, we introduce `commit()` method in `RecordContext()` just only
>> to
>>> call ProcessorContext.commit() inside. (see the above code snippet)
>>> So, in Rich interfaces, we are not dealing with  `ProcessorRecordContext`
>>> at all, and we leave all its methods as it is.
>>> In this KIP, we made `RecordContext` to be the parent class of
>>> `ProcessorRecordContext`, just because of they share quite amount of
>>> methods and it is 

[GitHub] kafka pull request #4145: KAFKA-4928: Add integration test for DumpLogSegmen...

[Sammers21]

GitHub user Sammers21 opened a pull request:

https://github.com/apache/kafka/pull/4145

KAFKA-4928: Add integration test for DumpLogSegments

Adding tests for `kafka.tools.DumpLogSegments`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sammers21/kafka test_for_dump_log_segments

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4145






---

Re: [DISCUSS] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Ismael Juma]

Seems sensible to me.

Ismael

On Thu, Oct 26, 2017 at 6:37 AM, Onur Karaman 
wrote:

> Hey everyone.
>
> Giving this another shot since it looks like there was a KIP number
> collision on the wiki page.
>
> I made a config kip, KIP-214: Add zookeeper.max.in.flight.requests config
> to the broker:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker
>
> Comments are welcome.
>
> - Onur
>

Re: Use self contained tokens instead of ACL

[Sönke Liebau]

Hi Manikumar,

I've looked over the KIP and had a quick look at the code in the PR as
well. In principle I think this would help Peter along depending on how
plugable some of the components are. Since Peter wants to generate Tokens
not in Kafka but in an external System the entire part in Kafka of
generating DelegationTokens would simply not be used, which I think would
be fine. To validate externally generated tokens an option to substitute
for example the TokenCache for a custom implementation or/and substitute
the method of authenticating a delegation token for a custom class.

Apologies for asking questions I could look up in the code myself, but at a
first glance I haven't seen any indications of this token system being
extendable, do you plan to allow extending the system to different external
token providers? OAuth would come to mind as a fairly wide spread candidate
that could probably be implemented fairly easily.

Kind regards,
Sönke

On Fri, Oct 27, 2017 at 11:17 AM, Manikumar 
wrote:

> Hi,
>
> We have a accepted KIP for adding delegation token support for Kafka.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 48+Delegation+token+support+for+Kafka
>
> currently the PR is under review process. Maybe this can used be as
> starting point for your requirement.
>
> https://github.com/apache/kafka/pull/3616
>
>
>
> On Fri, Oct 27, 2017 at 2:34 PM, Sönke Liebau <
> soenke.lie...@opencore.com.invalid> wrote:
>
> > Hi Peter,
> >
> > thanks for the explanation, it all makes sense now :)
> >
> > I can't say that I immediately see an easy way forward though to be
> honest.
> > The big issue, I think, is getting the token to Kafka (and hopefully
> there
> > is an easy way that I simply don't know of and someone will correct me) -
> > implementing a custom principalbuilder and authorizer should be almost
> > trivial.
> >
> > If transmitting the token as part of the ssl certificate or a Kerberos
> > ticket is out though the air gets a bit thin if you don't want to
> maintain
> > your own fork of Kafka. The only potential solution that I can come up
> with
> > is to piggyback on SASL and provide your own LoginModule in Kafka's jaas
> > file. If you use the SASL_SSL endpoint certificate checking should still
> > have occured before the SASL handshake is initialized, so you
> authenticated
> > the user at that point. You could then use that handshake to transmit
> your
> > token, have your custom principalbuilder extract the topics from that and
> > your custom authorizer authorize based on the extracted topicnames.
> > A word of caution though: this is based on a few minutes looking at code
> > and my dangerous half knowledge of SASL, so there are any number of
> things
> > that could make this impossible, either with SASL or in the Kafka
> codebase
> > itself. Might be a direction to explore though.
> >
> > Hopefully that makes sense and is targeted at least in the vicinity of
> whet
> > you are looking for?
> >
> > Kind regards,
> > Sönke
> >
> > On Fri, Oct 27, 2017 at 9:33 AM, Postmann, P. (Peter) <
> > peter.postm...@ing.com.invalid> wrote:
> >
> > > Hi Sönke,
> > >
> > > Thanks for your feedback, sorry that I didn’t gave you the whole
> picture
> > > in first place:
> > >
> > > We are using an Architecture, which tries to avoid to fetch or pull
> > > anything from a 3rd party during runtime. Therefore we are using
> > > self-contained tokens and client side load balancing with a micro
> service
> > > alike architecture.
> > >
> > > In this architecture we have two tokens:
> > > - the manifest which enabled services to provide APIs
> > > - the peer token which enables services to call APIs
> > >
> > > API providers publish their APIs in a Portal. API consumers subscribe
> to
> > > those APIs. The portal generates a manifest for the provider and a peer
> > > token for the consumer. Both tokens contain a list of endpoints and are
> > > signed by the portal. The tokens are valid for a certain amount of
> time.
> > >
> > > Furthermore we use a Service Registry to discover those services. The
> > flow
> > > works as follows:
> > >
> > > - A service instance registers itself (endpoint --> IP:Port) at the
> > > Service Discovery (SD) using the manifest.
> > > - A client queries SD for instances of a specific Endpoint and
> receives a
> > > list of IP:Port combinations
> > > - The client connects to the service and provides its peer token
> > >
> > > The client and the service use mutual TLS for authentication and the
> peer
> > > token for authorization. The token is signed to check its integrity and
> > > linked to the CN of the client certificate to check its validity (and
> > > prevent forwarding of the token).
> > >
> > > The benefit is, that we do not have any critical runtime dependencies:
> SD
> > > results can be cached and the tokens are valid for a certain amount of
> > > time. A client can perform client side load balancing and call a
> service
> > > even if the 

[GitHub] kafka pull request #4144: MINOR: Ensure that the producer in testAlterReplic...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4144


---

Build failed in Jenkins: kafka-trunk-jdk8 #2175

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Document "high watermark" magic value for delete records request

[ismael] MINOR: Update docs wrt topic deletion being enabled by default

--
[...truncated 384.65 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 

[jira] [Resolved] (KAFKA-6133) NullPointerException in S3 Connector when using rotate.interval.ms

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-6133.
--
Resolution: Invalid

Closing this as INVALID, since this is not Apache Kafka code. These 
connector-specific issues should be logged in the connector's project, which 
you've done 
[here|https://github.com/confluentinc/kafka-connect-storage-cloud/issues/109].

> NullPointerException in S3 Connector when using rotate.interval.ms
> --
>
> Key: KAFKA-6133
> URL: https://issues.apache.org/jira/browse/KAFKA-6133
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Elizabeth Bennett
>
> I just tried out the new rotate.interval.ms feature in the S3 connector to do 
> time based flushing. I am getting this NPE on every event:
> {code}[2017-10-20 23:21:35,233] ERROR Task foo-to-S3-0 threw an uncaught and 
> unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask)
> java.lang.NullPointerException
> at 
> io.confluent.connect.s3.TopicPartitionWriter.rotateOnTime(TopicPartitionWriter.java:288)
> at 
> io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:234)
> at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:180)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:464)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:265)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
> at 
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at 
> org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> [2017-10-20 23:21:35,233] ERROR Task is being killed and will not recover 
> until manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask)
> [2017-10-20 23:21:35,233] ERROR Task foo-to-S3-0 threw an uncaught and 
> unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due 
> to unrecoverable exception.
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:484)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:265)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
> at 
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
> at 
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at 
> org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748){code}
> I dug into the S3 connect code a bit and it looks like the 
> {{rotate.interval.ms}} feature only works if you are using the 
> TimeBasedPartitioner. It will get the TimestampExtractor class from the 
> TimeBasedPartitioner to determine the timestamp of the event, and will use 
> this for the time based flushing.
> I'm using a custom partitioner, but I'd still really like to use the 
> {{rotate.interval.ms}} feature, using wall clock time to determine the 
> flushing behavior.
> I'd be willing to work on fixing this issue, but I want to confirm it is 
> actually bug, and not that it was specifically designed to only work with the 
> TimeBasedPartitioner. Even if it is the latter, it should probably not crash 
> with an NPE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-1.0-jdk7 #62

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Document "high watermark" magic value for delete records request

--
[...truncated 1.82 MB...]

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns PASSED

[GitHub] kafka pull request #3951: MINOR: update listener.security.protocol.map confi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3951


---

Build failed in Jenkins: kafka-trunk-jdk8 #2174

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Fix indentation in KafkaApis.handleOffsetFetchRequest

--
[...truncated 386.48 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 

Build failed in Jenkins: kafka-trunk-jdk9 #156

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Fix indentation in KafkaApis.handleOffsetFetchRequest

[ismael] MINOR: Document "high watermark" magic value for delete records request

[ismael] MINOR: Update docs wrt topic deletion being enabled by default

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (couchdbtest ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 9b53c31d0854f02139acbd39c33f17ac4886914d 
(refs/remotes/origin/trunk)
Commit message: "MINOR: Update docs wrt topic deletion being enabled by default"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9b53c31d0854f02139acbd39c33f17ac4886914d
 > git rev-list 501a5e262702bcc043724cb9e1f536e16a66399e # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins6532495605010988447.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

ERROR: JAVA_HOME is set to an invalid directory: 
/home/jenkins/tools/java/latest1.9

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 1 mo 4 days old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user ism...@juma.me.uk

[GitHub] kafka pull request #3835: MINOR: update operations doc on topic deletion

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3835


---

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Jeyhun Karimov]

Hi Matthias,

Thanks for your comments. I agree that this is not the best way to do. A
bit of history behind this design.

Prior doing this, I tried to provide ProcessorContext itself as an argument
in Rich interfaces. However, we dont want to give users that flexibility
and “power”. Moreover, ProcessorContext contains processor level
information and not Record level info. The only thing we need ij
ProcessorContext is commit() method.

So, as far as I understood, we need recor context (offset, timestamp and
etc) but also we need a commit() method ( we dont want to provide
ProcessorContext as a parameter so users can use ProcessorContext.commit()
).

As a result, I thought to “propagate” commit() call from RecordContext to
ProcessorContext() .


If there is a misunderstanding in motvation/discussion of KIP/included
jiras please let me know.


Cheers,
Jeyhun


On Fri 27. Oct 2017 at 12:39, Matthias J. Sax  wrote:

> I am personally still not convinced, that we should add `commit()` at all.
>
> @Guozhang: you created the original Jira. Can you elaborate a little
> bit? Isn't requesting commits a low level API that should not be exposed
> in the DSL? Just want to understand the motivation better. Why would
> anybody that uses the DSL ever want to request a commit? To me,
> requesting commits is useful if you manipulated state explicitly, ie,
> via Processor API.
>
> Also, for the solution: it seem rather unnatural to me, that we add
> `commit()` to `RecordContext` -- from my understanding, `RecordContext`
> is an helper object that provide access to record meta data. Requesting
> a commit is something quite different. Additionally, a commit does not
> commit a specific record but a `RecrodContext` is for a specific record.
>
> To me, this does not seem to be a sound API design if we follow this path.
>
>
> -Matthias
>
>
>
> On 10/26/17 10:41 PM, Jeyhun Karimov wrote:
> > Hi,
> >
> > Thanks for your suggestions.
> >
> > I have some comments, to make sure that there is no misunderstanding.
> >
> >
> > 1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
> >> user to consolidate this call as
> >> "processorContext.recordContext().commit()". And internal implementation
> >> of
> >> `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
> >> this call.
> >
> >
> > - I think we should not deprecate `ProcessorContext.commit()`. The main
> > intuition that we introduce `commit()` in `RecordContext` is that,
> > `RecordContext` is the one which is provided in Rich interfaces. So if
> user
> > wants to commit, then there should be some method inside `RecordContext`
> to
> > do so. Internally, `RecordContext.commit()` calls
> > `ProcessorContext.commit()`  (see the last code snippet in KIP-159):
> >
> > @Override
> > public void process(final K1 key, final V1 value) {
> >
> > recordContext = new RecordContext() {   //
> > recordContext initialization is added in this KIP
> > @Override
> > public void commit() {
> > context().commit();
> > }
> >
> > @Override
> > public long offset() {
> > return context().recordContext().offset();
> > }
> >
> > @Override
> > public long timestamp() {
> > return context().recordContext().timestamp();
> > }
> >
> > @Override
> > public String topic() {
> > return context().recordContext().topic();
> > }
> >
> > @Override
> > public int partition() {
> > return context().recordContext().partition();
> > }
> >   };
> >
> >
> > So, we cannot deprecate `ProcessorContext.commit()` in this case IMO.
> >
> >
> > 2. Add the `task` reference to the impl class, `ProcessorRecordContext`,
> so
> >> that it can implement the commit call itself.
> >
> >
> > - Actually, I don't think that we need `commit()` in
> > `ProcessorRecordContext`. The main intuition is to "transfer"
> > `ProcessorContext.commit()` call to Rich interfaces, to support
> > user-specific committing.
> >  To do so, we introduce `commit()` method in `RecordContext()` just only
> to
> > call ProcessorContext.commit() inside. (see the above code snippet)
> > So, in Rich interfaces, we are not dealing with  `ProcessorRecordContext`
> > at all, and we leave all its methods as it is.
> > In this KIP, we made `RecordContext` to be the parent class of
> > `ProcessorRecordContext`, just because of they share quite amount of
> > methods and it is logical to enable inheritance between those two.
> >
> > 3. In the wiki page, the statement that "However, call to a commit()
> method,
> >> is valid only within RecordContext interface (at least for now), we
> throw
> >> an exception in ProcessorRecordContext.commit()." and the code snippet
> >> below would need to be updated as well.
> >
> >
> > - I think 

[GitHub] kafka pull request #4119: MINOR: added -1 value description as "high waterma...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4119


---

[GitHub] kafka pull request #4139: MINOR: Indentation Fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4139


---

[GitHub] kafka pull request #4144: MINOR: Ensure that the producer in testAlterReplic...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4144

MINOR: Ensure that the producer in testAlterReplicaLogDirs is always closed

Failure to close the producer could cause a transient failure, more details
below.

The request timeout was only 2 seconds, exceptions thrown were not
propagated and the producer would not be closed. If the exception
was thrown during `send`, we did not increment `numMessages`
allowing the test to pass.

I have increased the timeout to 10 seconds and made sure that
exceptions are propagated.

Example of the error:

```text
kafka.api.SaslSslAdminClientIntegrationTest > classMethod STARTED

kafka.api.SaslSslAdminClientIntegrationTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads, 
allThreads=Set(metrics-meter-tick-thread-2, Signal Dispatcher, main, Reference 
Handler, scala-execution-context-global-164, kafka-producer-network-thread | 
producer-1, scala-execution-context-global-166, Test worker, 
scala-execution-context-global-1249, /0:0:0:0:0:0:0:1:58910 to 
/0:0:0:0:0:0:0:1:43025 workers Thread 2, Finalizer, /0:0:0:0:0:0:0:1:58910 to 
/0:0:0:0:0:0:0:1:43025 workers Thread 3, scala-execution-context-global-163, 
metrics-meter-tick-thread-1)
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
ensure-producer-is-closed-test-alter-replica-log-dirs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4144


commit 11153a454cd7c561202ce3b86891715a3670b2c6
Author: Ismael Juma 
Date:   2017-10-27T12:58:59Z

MINOR: Ensure that the producer in testAlterReplicaLogDirs is always closed

The request timeout was only 2 seconds and if an exception was thrown,
it would not be propagated and the producer would be leaked. If the 
exception
was thrown during `send`, we would not increment the counter and the
test would pass.

Increased the timeout to 10 seconds and made sure that a producer
exception is now propagated.




---

[jira] [Created] (KAFKA-6136) Transient test failure: SaslPlainSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6136:
--

 Summary: Transient test failure: 
SaslPlainSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials
 Key: KAFKA-6136
 URL: https://issues.apache.org/jira/browse/KAFKA-6136
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma


Looks like a cleanup issue:

{code}
testTwoConsumersWithDifferentSaslCredentials – 
kafka.api.SaslPlainSslEndToEndAuthorizationTest
a few seconds
Error
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to 
access group: group
Stacktrace
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to 
access group: group
Standard Output
[2017-10-27 00:37:47,919] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
Adding ACLs for resource `Cluster:kafka-cluster`: 
User:admin has Allow permission for operations: ClusterAction from 
hosts: * 
Current ACLs for resource `Cluster:kafka-cluster`: 
User:admin has Allow permission for operations: ClusterAction from 
hosts: * 
[2017-10-27 00:37:48,961] ERROR [ReplicaFetcher replicaId=2, leaderId=0, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 0 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
[2017-10-27 00:37:48,967] ERROR [ReplicaFetcher replicaId=1, leaderId=0, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 0 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
Adding ACLs for resource `Topic:*`: 
User:admin has Allow permission for operations: Read from hosts: * 
Current ACLs for resource `Topic:*`: 
User:admin has Allow permission for operations: Read from hosts: * 
[2017-10-27 00:37:52,330] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
[2017-10-27 00:37:52,345] ERROR ZKShutdownHandler is not registered, so 
ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
changes (org.apache.zookeeper.server.ZooKeeperServer:472)
Adding ACLs for resource `Cluster:kafka-cluster`: 
User:admin has Allow permission for operations: ClusterAction from 
hosts: * 
Current ACLs for resource `Cluster:kafka-cluster`: 
User:admin has Allow permission for operations: ClusterAction from 
hosts: * 
[2017-10-27 00:37:53,459] ERROR [ReplicaFetcher replicaId=1, leaderId=0, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 0 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
[2017-10-27 00:37:53,462] ERROR [ReplicaFetcher replicaId=2, leaderId=0, 
fetcherId=0] Error for partition __consumer_offsets-0 from broker 0 
(kafka.server.ReplicaFetcherThread:107)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [Topic authorization failed.]
Adding ACLs for resource `Topic:*`: 
User:admin has Allow permission for operations: Read from hosts: * 
Current ACLs for resource `Topic:*`: 
User:admin has Allow permission for operations: Read from hosts: * 
Adding ACLs for resource `Topic:e2etopic`: 
User:user has Allow permission for operations: Write from hosts: *
User:user has Allow permission for operations: Describe from hosts: * 
Adding ACLs for resource `Cluster:kafka-cluster`: 
User:user has Allow permission for operations: Create from hosts: * 
Current ACLs for resource `Topic:e2etopic`: 
User:user has Allow permission for operations: Write from hosts: *
User:user has Allow permission for operations: Describe from hosts: * 
Adding ACLs for resource `Topic:e2etopic`: 
User:user has Allow permission for operations: Read from hosts: *
User:user has Allow permission for operations: Describe from hosts: * 
Adding ACLs for resource `Group:group`: 
User:user has Allow permission for operations: Read from hosts: * 
Current ACLs for resource `Topic:e2etopic`: 
User:user has Allow permission for operations: Write from hosts: *
User:user has Allow permission for operations: Describe from hosts: *
User:user has Allow permission for operations: Read from hosts: * 
Current ACLs for resource `Group:group`: 
User:user has Allow permission for operations: Read from hosts: * 
[2017-10-27 00:37:55,520] WARN caught end of stream exception 
(org.apache.zookeeper.server.NIOServerCnxn:368)
EndOfStreamException: Unable to read 

Re: [DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Jeff Klukas]

I added a note in the KIP about ConfigException being thrown. I also
changed the proposed default for the new config to empty string rather than
null.

Absent a clear definition of what "common" regex syntax is, it seems an
undue burden to ask the user to guess at what Pattern features are safe. If
we do end up implementing a different regex style, I think it will be
necessary to still support the Java Pattern style long-term as an option.
If we want to use a different regex style as default down the road, we
could require "power users" of Java Pattern to enable an additional config
option to maintain compatibility.

One additional change I might make to the KIP is that 'topics.regex' might
be a better choice for config name than 'topics.pattern'. That would be in
keeping with RegexRouter that has a 'regex' configuration option rather
than 'pattern'.

On Thu, Oct 26, 2017 at 11:00 PM, Ewen Cheslack-Postava 
wrote:

> It's fine to be more detailed, but ConfigException is already implied for
> all other config issues as well.
>
> Default could be either null or just empty string. re: alternatives, if you
> wanted to be slightly more detailed (though still a bit vague) re:
> supported syntax, you could just say that while Pattern is used, we only
> guarantee support for common regular expression syntax. Not sure if there's
> a good way of defining what "common" syntax is.
>
> Otherwise LGTM, and thanks for helping fill in a longstanding gap!
>
> -Ewen
>
> On Thu, Oct 26, 2017 at 7:56 PM, Ted Yu  wrote:
>
> > bq. Users may specify only one of 'topics' or 'topics.pattern'.
> >
> > Can you fill in which exception would be thrown if both of them are
> > specified
> > ?
> >
> > Cheers
> >
> > On Thu, Oct 26, 2017 at 6:27 PM, Jeff Klukas  wrote:
> >
> > > Looking for feedback on
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > >
> >
>

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Matthias J. Sax]

I am personally still not convinced, that we should add `commit()` at all.

@Guozhang: you created the original Jira. Can you elaborate a little
bit? Isn't requesting commits a low level API that should not be exposed
in the DSL? Just want to understand the motivation better. Why would
anybody that uses the DSL ever want to request a commit? To me,
requesting commits is useful if you manipulated state explicitly, ie,
via Processor API.

Also, for the solution: it seem rather unnatural to me, that we add
`commit()` to `RecordContext` -- from my understanding, `RecordContext`
is an helper object that provide access to record meta data. Requesting
a commit is something quite different. Additionally, a commit does not
commit a specific record but a `RecrodContext` is for a specific record.

To me, this does not seem to be a sound API design if we follow this path.


-Matthias



On 10/26/17 10:41 PM, Jeyhun Karimov wrote:
> Hi,
> 
> Thanks for your suggestions.
> 
> I have some comments, to make sure that there is no misunderstanding.
> 
> 
> 1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
>> user to consolidate this call as
>> "processorContext.recordContext().commit()". And internal implementation
>> of
>> `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
>> this call.
> 
> 
> - I think we should not deprecate `ProcessorContext.commit()`. The main
> intuition that we introduce `commit()` in `RecordContext` is that,
> `RecordContext` is the one which is provided in Rich interfaces. So if user
> wants to commit, then there should be some method inside `RecordContext` to
> do so. Internally, `RecordContext.commit()` calls
> `ProcessorContext.commit()`  (see the last code snippet in KIP-159):
> 
> @Override
> public void process(final K1 key, final V1 value) {
> 
> recordContext = new RecordContext() {   //
> recordContext initialization is added in this KIP
> @Override
> public void commit() {
> context().commit();
> }
> 
> @Override
> public long offset() {
> return context().recordContext().offset();
> }
> 
> @Override
> public long timestamp() {
> return context().recordContext().timestamp();
> }
> 
> @Override
> public String topic() {
> return context().recordContext().topic();
> }
> 
> @Override
> public int partition() {
> return context().recordContext().partition();
> }
>   };
> 
> 
> So, we cannot deprecate `ProcessorContext.commit()` in this case IMO.
> 
> 
> 2. Add the `task` reference to the impl class, `ProcessorRecordContext`, so
>> that it can implement the commit call itself.
> 
> 
> - Actually, I don't think that we need `commit()` in
> `ProcessorRecordContext`. The main intuition is to "transfer"
> `ProcessorContext.commit()` call to Rich interfaces, to support
> user-specific committing.
>  To do so, we introduce `commit()` method in `RecordContext()` just only to
> call ProcessorContext.commit() inside. (see the above code snippet)
> So, in Rich interfaces, we are not dealing with  `ProcessorRecordContext`
> at all, and we leave all its methods as it is.
> In this KIP, we made `RecordContext` to be the parent class of
> `ProcessorRecordContext`, just because of they share quite amount of
> methods and it is logical to enable inheritance between those two.
> 
> 3. In the wiki page, the statement that "However, call to a commit() method,
>> is valid only within RecordContext interface (at least for now), we throw
>> an exception in ProcessorRecordContext.commit()." and the code snippet
>> below would need to be updated as well.
> 
> 
> - I think above explanation covers this as well.
> 
> 
> I want to gain some speed to this KIP, as it has gone though many changes
> based on user/developer needs, both in documentation-/implementation-wise.
> 
> 
> Cheers,
> Jeyhun
> 
> 
> 
> On Tue, Oct 24, 2017 at 1:41 AM Guozhang Wang  wrote:
> 
>> Thanks for the information Jeyhun. I had also forgot about KAFKA-3907 with
>> this KIP..
>>
>> Thinking a bit more, I'm now inclined to go with what we agreed before, to
>> add the commit() call to `RecordContext`. A few minor tweaks on its
>> implementation:
>>
>> 1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
>> user to consolidate this call as
>> "processorContext.recordContext().commit()". And internal implementation
>> of
>> `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
>> this call.
>>
>> 2. Add the `task` reference to the impl class, `ProcessorRecordContext`, so
>> that it can implement the commit call itself.
>>
>> 3. In the wiki page, the statement that "However, call to a commit()
>> method,
>> is valid only within RecordContext interface (at least 

[GitHub] kafka pull request #4143: MINOR: PartitionReassignmentHandler should generat...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4143

MINOR: PartitionReassignmentHandler should generate event only when znode 
is created and eliminate unnecessary allocations

As suggested by Jun in https://github.com/apache/kafka/pull/4141, we
only need to generate the event when the znode is created (
controller failover restarts the reassignment without generating
an event).

I also took the chance to eliminate all the unnecessary allocations
of `TopicPartition` and `TopicAndPartition`. We now use the former
in the Controller (bringing it inline with the rest of the non legacy
code).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
partition-reassignment-ignore-handle-deletion-and-data-change

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4143


commit f2a81b5ef493e428d93be3340148f101089d87e3
Author: Ismael Juma 
Date:   2017-10-27T09:46:02Z

Don't create PartitionReassignment event if the znode is deleted or changed

commit 9f448fda833db54e2d558fc69f0e882f9262fab6
Author: Ismael Juma 
Date:   2017-10-27T10:25:34Z

Use TopicPartition in Controller and avoid allocations




---

Re: Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Ted Yu]

I think if you explain what A and B are in the beginning, it makes sense to
use them since readers would know who they reference.

Cheers

On Thu, Oct 26, 2017 at 11:04 PM, Jan Filipiak 
wrote:

>
>
> Thanks for the remarks. hope I didn't miss any.
> Not even sure if it makes sense to introduce A and B or just stick with
> "this ktable", "other ktable"
>
> Thank you
> Jan
>
>
> On 27.10.2017 06:58, Ted Yu wrote:
>
>> Do you mind addressing my previous comments ?
>>
>> http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+
>> DISCUSS+KIP+213+Support+non+key+joining+in+KTable
>>
>> On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak 
>> wrote:
>>
>> Hello everyone,
>>>
>>> this is the new discussion thread after the ID-clash.
>>>
>>> Best
>>> Jan
>>>
>>> __
>>>
>>>
>>> Hello Kafka-users,
>>>
>>> I want to continue with the development of KAFKA-3705, which allows the
>>> Streams DSL to perform KTableKTable-Joins when the KTables have a
>>> one-to-many relationship.
>>> To make sure we cover the requirements of as many users as possible and
>>> have a good solution afterwards I invite everyone to read through the
>>> KIP I
>>> put together and discuss it here in this Thread.
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
>>> Support+non-key+joining+in+KTable
>>> https://issues.apache.org/jira/browse/KAFKA-3705
>>> https://github.com/apache/kafka/pull/3720
>>>
>>> I think a public discussion and vote on a solution is exactly what is
>>> needed to bring this feauture into kafka-streams. I am looking forward to
>>> everyones opinion!
>>>
>>> Please keep the discussion on the mailing list rather than commenting on
>>> the wiki (wiki discussions get unwieldy fast).
>>>
>>> Best
>>> Jan
>>>
>>>
>>>
>>>
>
>
>

Re: Use self contained tokens instead of ACL

[Manikumar]

Hi,

We have a accepted KIP for adding delegation token support for Kafka.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka

currently the PR is under review process. Maybe this can used be as
starting point for your requirement.

https://github.com/apache/kafka/pull/3616



On Fri, Oct 27, 2017 at 2:34 PM, Sönke Liebau <
soenke.lie...@opencore.com.invalid> wrote:

> Hi Peter,
>
> thanks for the explanation, it all makes sense now :)
>
> I can't say that I immediately see an easy way forward though to be honest.
> The big issue, I think, is getting the token to Kafka (and hopefully there
> is an easy way that I simply don't know of and someone will correct me) -
> implementing a custom principalbuilder and authorizer should be almost
> trivial.
>
> If transmitting the token as part of the ssl certificate or a Kerberos
> ticket is out though the air gets a bit thin if you don't want to maintain
> your own fork of Kafka. The only potential solution that I can come up with
> is to piggyback on SASL and provide your own LoginModule in Kafka's jaas
> file. If you use the SASL_SSL endpoint certificate checking should still
> have occured before the SASL handshake is initialized, so you authenticated
> the user at that point. You could then use that handshake to transmit your
> token, have your custom principalbuilder extract the topics from that and
> your custom authorizer authorize based on the extracted topicnames.
> A word of caution though: this is based on a few minutes looking at code
> and my dangerous half knowledge of SASL, so there are any number of things
> that could make this impossible, either with SASL or in the Kafka codebase
> itself. Might be a direction to explore though.
>
> Hopefully that makes sense and is targeted at least in the vicinity of whet
> you are looking for?
>
> Kind regards,
> Sönke
>
> On Fri, Oct 27, 2017 at 9:33 AM, Postmann, P. (Peter) <
> peter.postm...@ing.com.invalid> wrote:
>
> > Hi Sönke,
> >
> > Thanks for your feedback, sorry that I didn’t gave you the whole picture
> > in first place:
> >
> > We are using an Architecture, which tries to avoid to fetch or pull
> > anything from a 3rd party during runtime. Therefore we are using
> > self-contained tokens and client side load balancing with a micro service
> > alike architecture.
> >
> > In this architecture we have two tokens:
> > - the manifest which enabled services to provide APIs
> > - the peer token which enables services to call APIs
> >
> > API providers publish their APIs in a Portal. API consumers subscribe to
> > those APIs. The portal generates a manifest for the provider and a peer
> > token for the consumer. Both tokens contain a list of endpoints and are
> > signed by the portal. The tokens are valid for a certain amount of time.
> >
> > Furthermore we use a Service Registry to discover those services. The
> flow
> > works as follows:
> >
> > - A service instance registers itself (endpoint --> IP:Port) at the
> > Service Discovery (SD) using the manifest.
> > - A client queries SD for instances of a specific Endpoint and receives a
> > list of IP:Port combinations
> > - The client connects to the service and provides its peer token
> >
> > The client and the service use mutual TLS for authentication and the peer
> > token for authorization. The token is signed to check its integrity and
> > linked to the CN of the client certificate to check its validity (and
> > prevent forwarding of the token).
> >
> > The benefit is, that we do not have any critical runtime dependencies: SD
> > results can be cached and the tokens are valid for a certain amount of
> > time. A client can perform client side load balancing and call a service
> > even if the SD or the Portal are currently unavailable. Furthermore we
> > avoid bottlenecks like load balancers.
> >
> > If you have 20mins time, our Principal Architect for API Design gave a
> > talk about this: https://www.youtube.com/watch?v=Yke6Vut2Shc
> >
> > We want to use the same mechanism with Kafka:
> > - Connect via Mutual TLS
> > - Provide a peer token which contains metod:topics
> >
> > I understood that it should be possible to get the details from the
> > certificate, but we also need the token. The combination of certificate +
> > token is need to prevent impersonation of APIs (one could easily forward
> a
> > token, if it wasn’t bound to a certificate).
> >
> > I agree with the revocation part, but we are only looking at internal
> > clients. The process would be to revoke access in the Portal which
> prevents
> > new instances to connect to the service. If we really encounter malicious
> > behaviour, the prefer way is to shut down the misbehaving client.
> >
> > Kind Regards,
> > Peter
> >
> >
> > -Original Message-
> > From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID]
> > Sent: Donnerstag, 26. Oktober 2017 14:59
> > To: dev@kafka.apache.org
> > Subject: Re: Use self contained tokens instead of 

Re: Use self contained tokens instead of ACL

[Sönke Liebau]

Hi Peter,

thanks for the explanation, it all makes sense now :)

I can't say that I immediately see an easy way forward though to be honest.
The big issue, I think, is getting the token to Kafka (and hopefully there
is an easy way that I simply don't know of and someone will correct me) -
implementing a custom principalbuilder and authorizer should be almost
trivial.

If transmitting the token as part of the ssl certificate or a Kerberos
ticket is out though the air gets a bit thin if you don't want to maintain
your own fork of Kafka. The only potential solution that I can come up with
is to piggyback on SASL and provide your own LoginModule in Kafka's jaas
file. If you use the SASL_SSL endpoint certificate checking should still
have occured before the SASL handshake is initialized, so you authenticated
the user at that point. You could then use that handshake to transmit your
token, have your custom principalbuilder extract the topics from that and
your custom authorizer authorize based on the extracted topicnames.
A word of caution though: this is based on a few minutes looking at code
and my dangerous half knowledge of SASL, so there are any number of things
that could make this impossible, either with SASL or in the Kafka codebase
itself. Might be a direction to explore though.

Hopefully that makes sense and is targeted at least in the vicinity of whet
you are looking for?

Kind regards,
Sönke

On Fri, Oct 27, 2017 at 9:33 AM, Postmann, P. (Peter) <
peter.postm...@ing.com.invalid> wrote:

> Hi Sönke,
>
> Thanks for your feedback, sorry that I didn’t gave you the whole picture
> in first place:
>
> We are using an Architecture, which tries to avoid to fetch or pull
> anything from a 3rd party during runtime. Therefore we are using
> self-contained tokens and client side load balancing with a micro service
> alike architecture.
>
> In this architecture we have two tokens:
> - the manifest which enabled services to provide APIs
> - the peer token which enables services to call APIs
>
> API providers publish their APIs in a Portal. API consumers subscribe to
> those APIs. The portal generates a manifest for the provider and a peer
> token for the consumer. Both tokens contain a list of endpoints and are
> signed by the portal. The tokens are valid for a certain amount of time.
>
> Furthermore we use a Service Registry to discover those services. The flow
> works as follows:
>
> - A service instance registers itself (endpoint --> IP:Port) at the
> Service Discovery (SD) using the manifest.
> - A client queries SD for instances of a specific Endpoint and receives a
> list of IP:Port combinations
> - The client connects to the service and provides its peer token
>
> The client and the service use mutual TLS for authentication and the peer
> token for authorization. The token is signed to check its integrity and
> linked to the CN of the client certificate to check its validity (and
> prevent forwarding of the token).
>
> The benefit is, that we do not have any critical runtime dependencies: SD
> results can be cached and the tokens are valid for a certain amount of
> time. A client can perform client side load balancing and call a service
> even if the SD or the Portal are currently unavailable. Furthermore we
> avoid bottlenecks like load balancers.
>
> If you have 20mins time, our Principal Architect for API Design gave a
> talk about this: https://www.youtube.com/watch?v=Yke6Vut2Shc
>
> We want to use the same mechanism with Kafka:
> - Connect via Mutual TLS
> - Provide a peer token which contains metod:topics
>
> I understood that it should be possible to get the details from the
> certificate, but we also need the token. The combination of certificate +
> token is need to prevent impersonation of APIs (one could easily forward a
> token, if it wasn’t bound to a certificate).
>
> I agree with the revocation part, but we are only looking at internal
> clients. The process would be to revoke access in the Portal which prevents
> new instances to connect to the service. If we really encounter malicious
> behaviour, the prefer way is to shut down the misbehaving client.
>
> Kind Regards,
> Peter
>
>
> -Original Message-
> From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID]
> Sent: Donnerstag, 26. Oktober 2017 14:59
> To: dev@kafka.apache.org
> Subject: Re: Use self contained tokens instead of ACL
>
> Hi Peter,
>
> I am not entirely sure that I understand what you mean when you say "at
> application level" to be honest, but I do understand that you want to
> forego 3rd party tools. However, this would mean that you have to implement
> some form of trust between your portal which issues the tokens and Kafka
> which checks the tokens. I am not sure that I'd recommend rolling your own
> solution here, authentication has a lot of pitfalls that can turn around to
> bite you.
> Same thing as what I proposed with Kerberos could be done via certificates
> of course. If your user requests access 

RE: Use self contained tokens instead of ACL

[Postmann, P. (Peter)]

Hi Sönke,

Thanks for your feedback, sorry that I didn’t gave you the whole picture in 
first place:

We are using an Architecture, which tries to avoid to fetch or pull anything 
from a 3rd party during runtime. Therefore we are using self-contained tokens 
and client side load balancing with a micro service alike architecture.

In this architecture we have two tokens:
- the manifest which enabled services to provide APIs
- the peer token which enables services to call APIs

API providers publish their APIs in a Portal. API consumers subscribe to those 
APIs. The portal generates a manifest for the provider and a peer token for the 
consumer. Both tokens contain a list of endpoints and are signed by the portal. 
The tokens are valid for a certain amount of time.

Furthermore we use a Service Registry to discover those services. The flow 
works as follows:

- A service instance registers itself (endpoint --> IP:Port) at the Service 
Discovery (SD) using the manifest.
- A client queries SD for instances of a specific Endpoint and receives a list 
of IP:Port combinations
- The client connects to the service and provides its peer token

The client and the service use mutual TLS for authentication and the peer token 
for authorization. The token is signed to check its integrity and linked to the 
CN of the client certificate to check its validity (and prevent forwarding of 
the token). 

The benefit is, that we do not have any critical runtime dependencies: SD 
results can be cached and the tokens are valid for a certain amount of time. A 
client can perform client side load balancing and call a service even if the SD 
or the Portal are currently unavailable. Furthermore we avoid bottlenecks like 
load balancers.

If you have 20mins time, our Principal Architect for API Design gave a talk 
about this: https://www.youtube.com/watch?v=Yke6Vut2Shc

We want to use the same mechanism with Kafka:
- Connect via Mutual TLS
- Provide a peer token which contains metod:topics

I understood that it should be possible to get the details from the 
certificate, but we also need the token. The combination of certificate + token 
is need to prevent impersonation of APIs (one could easily forward a token, if 
it wasn’t bound to a certificate). 

I agree with the revocation part, but we are only looking at internal clients. 
The process would be to revoke access in the Portal which prevents new 
instances to connect to the service. If we really encounter malicious 
behaviour, the prefer way is to shut down the misbehaving client.

Kind Regards,
Peter


-Original Message-
From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] 
Sent: Donnerstag, 26. Oktober 2017 14:59
To: dev@kafka.apache.org
Subject: Re: Use self contained tokens instead of ACL

Hi Peter,

I am not entirely sure that I understand what you mean when you say "at 
application level" to be honest, but I do understand that you want to forego 
3rd party tools. However, this would mean that you have to implement some form 
of trust between your portal which issues the tokens and Kafka which checks the 
tokens. I am not sure that I'd recommend rolling your own solution here, 
authentication has a lot of pitfalls that can turn around to bite you.
Same thing as what I proposed with Kerberos could be done via certificates of 
course. If your user requests access to a topic in the portal he uploads a 
certificate signing request with the topic name encoded in some field (lots of 
extensions to pick from). The portal signs the request and returns it to the 
user. When the user now connects to Kafka he does so using this certificate and 
it should be fairly easy for you to extend the KafkaPrincipalBuilder class and 
extract the list of these topics from the certificate. Then you'd need to also 
extend SimpleAclAuthorizer to check for these topics in the principal name and 
allow access if the topic is present in the certificate.

I am unsure of the benefits of this system over for example having the portal 
create ACLs in Kafka, when the user is granted access to a topic.
One scenario that comes to mind is if you dynamically spin up new clusters, if 
these new clusters all trust the same ca they would also accept these tokens 
right away, no need to create ACLs in fresh clusters. But this could also be 
solved by having a central repository for ACLs that are applied to all clusters 
regularly..
A drawback of this system is that you'd need to revoke certificates if you want 
to withdraw access to a topic, which is significantly harder that just deleting 
an ACL.

Anyway, not sure if this helps you at all, just some random musings - if you 
explain your specific needs a bit more we can discuss further.

Kind regards,
Sönke

On Wed, Oct 25, 2017 at 1:10 PM, Postmann, P. (Peter) < 
peter.postm...@ing.com.invalid> wrote:

> Hi Sönke,
>
> Thanks for the fast replay. We don’t want to use Kerberos since we 
> want to do the authorization on Application level and 

Build failed in Jenkins: kafka-trunk-jdk7 #2928

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6119: Bump epoch when expiring transactions in the

--
[...truncated 382.23 KB...]

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist STARTED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride PASSED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations STARTED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLock STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLock PASSED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
STARTED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedForMessageFormatOnHandleWriteTxnMarkersWhenMagicLowerThanRequired
 STARTED

kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedForMessageFormatOnHandleWriteTxnMarkersWhenMagicLowerThanRequired
 PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleTxnOffsetCommitRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleTxnOffsetCommitRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest STARTED

kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest PASSED

kafka.server.KafkaApisTest > 
shouldAppendToLogOnWriteTxnMarkersWhenCorrectMagicVersion STARTED

kafka.server.KafkaApisTest > 
shouldAppendToLogOnWriteTxnMarkersWhenCorrectMagicVersion PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleWriteTxnMarkersRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleWriteTxnMarkersRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicWhenPartitionIsNotHosted STARTED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicWhenPartitionIsNotHosted PASSED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetEarliestOffsetEqualsLastStableOffset STARTED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetEarliestOffsetEqualsLastStableOffset PASSED

kafka.server.KafkaApisTest > testReadCommittedConsumerListOffsetLatest STARTED

kafka.server.KafkaApisTest > testReadCommittedConsumerListOffsetLatest PASSED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetLimitedAtLastStableOffset STARTED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetLimitedAtLastStableOffset PASSED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetEarliestOffsetEqualsHighWatermark STARTED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetEarliestOffsetEqualsHighWatermark PASSED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetLimitedAtHighWatermark STARTED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetLimitedAtHighWatermark PASSED

[jira] [Resolved] (KAFKA-6132) KafkaProducer.initTransactions dead locks

[Piotr Nowojski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Nowojski resolved KAFKA-6132.
---
   Resolution: Duplicate
Fix Version/s: 1.0.0

This is probably a duplicate of  KAFKA-6042. Fixed in 1.0.0-rc3

> KafkaProducer.initTransactions dead locks
> -
>
> Key: KAFKA-6132
> URL: https://issues.apache.org/jira/browse/KAFKA-6132
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.11.0.0, 0.11.0.1
> Environment: Travis
>Reporter: Piotr Nowojski
>Priority: Critical
> Fix For: 1.0.0
>
>
> I have found some intermittent failures on travis when using Kafka 0.11 
> transactions for writing. One of them is a apparent deadlock with the 
> following stack trace:
> {code:java}
> "KafkaTestThread[19, false]" #166 prio=5 os_prio=0 tid=0x7f4b64c29800 
> nid=0x1260 waiting on condition [0x7f4b10fa4000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x947048a8> (a 
> java.util.concurrent.CountDownLatch$Sync)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>   at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:50)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:537)
> {code}
> I was unsuccessful to reproduce it locally, however I think I can semi 
> reliably reproduce it on Travis. Scenario includes simultaneous sequence of 
> instantiating new producers, calling {{KafkaProducer.initTransactions}}, 
> closing them interleaved with writing. I have created a stripped down version 
> of this scenario as a github project:
> https://github.com/pnowojski/kafka-init-deadlock
> The code for the test scenario is here:
> https://github.com/pnowojski/kafka-init-deadlock/blob/master/src/test/java/pl/nowojski/KafkaInitDeadLockTest.java
> I have defined 30 build profiles that run this test and in case of detecting 
> a dead lock (5 minutes period of inactivity), stack trace of all threads is 
> being printed out. Example travis run:
> https://travis-ci.org/pnowojski/kafka-init-deadlock/builds/293256284
> as you can see deadlock occurred in 7 out of 30 builds. It seems like in this 
> scenario all of them are failing/dead locking in exactly same way. 
> I have observed this issue both on 0.11.0.0 and 0.11.0.1 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4137: KAFKA-6119: Bump epoch when expiring transactions ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4137


---

Jenkins build is back to normal : kafka-1.0-jdk7 #60

[Apache Jenkins Server]

See 

Fwd: Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Jan Filipiak]

Sorry,

so used to the users mailing list...


Thanks for the remarks. hope I didn't miss any.
Not even sure if it makes sense to introduce A and B or just stick with
"this ktable", "other ktable"

Thank you
Jan


On 27.10.2017 06:58, Ted Yu wrote:

Do you mind addressing my previous comments ?

http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+DISCUSS+KIP+213+Support+non+key+joining+in+KTable

On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak 
wrote:


Hello everyone,

this is the new discussion thread after the ID-clash.

Best
Jan

__


Hello Kafka-users,

I want to continue with the development of KAFKA-3705, which allows the
Streams DSL to perform KTableKTable-Joins when the KTables have a
one-to-many relationship.
To make sure we cover the requirements of as many users as possible and
have a good solution afterwards I invite everyone to read through the KIP I
put together and discuss it here in this Thread.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
Support+non-key+joining+in+KTable
https://issues.apache.org/jira/browse/KAFKA-3705
https://github.com/apache/kafka/pull/3720

I think a public discussion and vote on a solution is exactly what is
needed to bring this feauture into kafka-streams. I am looking forward to
everyones opinion!

Please keep the discussion on the mailing list rather than commenting on
the wiki (wiki discussions get unwieldy fast).

Best
Jan