Re: [DISCUSS] KIP-82 - Add Record Headers

[Renu Tewari]

+1 Nacho, Radai, Mayuresh
1) yes for unordered keys. Given the expected number of headers is not that
high linear searching is ok. The overhead of ordering and the dependence on
client implementations makes it error prone.
2) yes on proposed key space. Adding structure makes it easier to manage
and delegate handling for different use cases. Also simplifies debugging
and special casing.

Thanks
Renu

On Thu, Oct 6, 2016 at 3:31 PM, Mayuresh Gharat 
wrote:

> +1 Nacho, Radai.
>
> Ordering the Keys would help if we were gonna look at the headers linearly
> but given the disadvantage that the client implementations have to know the
> order of headers in order that the reading system in the pipeline doesn't
> break, unordered list sounds better.
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, Oct 6, 2016 at 2:46 PM, Nacho Solis 
> wrote:
>
> > I'm also
> >
> > 1. no  (ordered keys)
> > 2. yes (propose key space)
> >
> >
> > 1. I don't think there is going to be much savings in ordering the keys.
> > I'm assuming some parsing will happen either way. Ordering the keys would
> > be useful if we were doing linear search on the headers, and even then,
> the
> > performance difference would be small for any reasonable number of
> headers
> > (even anything that fits in 1 meg).
> >
> > However, I think that it's likely that whoever is looking at the headers
> is
> > going to want to search for plugins for any header in existence, as such,
> > it's going to have to iterate over the whole header set. So, for every
> > header, look up the plugin, not the other way around.  Even if we did it
> > the other way around (for every plugin, search if there is a header
> > present) we would expect only to have an algorithm that is O(n) and only
> > iterate once over the list. We wouldn't need to iterate more than once.
> >
> > Given this, the code overhead of ordering the headers when something is
> > inserted and such is a bigger pain than dealing with a potentially
> > unordered list.
> >
> >
> > 2. I like structure and to reduce the play space for potential keys.
> This
> > will allow us to do filter and know when we're testing. At the same time,
> > we're reserving a lot of space for future usages. However, if there is no
> > agreement on this I don't think it would be a blocker.  I just want to
> make
> > sure we have some order and if possible contiguous ranges for similar
> > usages.
> >
> > Nacho
> >
> >
> > On Thu, Oct 6, 2016 at 2:26 PM, radai 
> wrote:
> >
> > > 1. tending towards no, but I dont have any strong opinions on header
> > > ordering. it offers a potential speedup for header lookup in a
> serialized
> > > blob (in wire format) but that goes away if the headers are fully
> > > serialized/deserialized always. on the downside its an implementation
> > > detail that 3rd party impls would need to worry about, and would be
> hard
> > to
> > > diagnose if they fail to. its also less friendly to high performance io
> > > (think about appending headers to an existing blob in pass-through
> > > components like mirror-maker vs write to the middle) - its still
> possible
> > > though. however, the kafka code base is far from being iovec friendly
> > > anyway.
> > >
> > > 2. yes.
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Oct 6, 2016 at 8:58 AM, K Burstev 
> wrote:
> > >
> > > > @Mayuresh
> > > >
> > > > Yes exactly, it is a real nasty race issue.
> > > >
> > > > This is why I look forward to being able to trash our custom
> workaround
> > > :)
> > > >
> > > > Kostya
> > > >
> > > >
> > > > 06.10.2016, 02:36, "Mayuresh Gharat" :
> > > > > @Kostya
> > > > >
> > > > > Regarding "To get around this we have an awful *cough* solution
> > whereby
> > > > we
> > > > > have to send our message wrapper with the headers and null content,
> > and
> > > > > then we have an application that has to consume from all the
> > compacted
> > > > > topics and when it sees this message it produces back in a null
> > payload
> > > > > record to make the broker compact it out."
> > > > >
> > > > >  ---> This has a race condition, right?
> > > > >
> > > > > Suppose the producer produces a message with headers and null
> content
> > > at
> > > > > time To to Kafka.
> > > > >
> > > > > Then the producer, at time To + 1, sends another message with
> headers
> > > and
> > > > > actual content to Kafka.
> > > > >
> > > > > What we expect is that the application that is consuming and then
> > > > producing
> > > > > same message with null payload should happen at time To + 0.5, so
> > that
> > > > the
> > > > > message at To + 1 is not deleted.
> > > > >
> > > > > But there is no guarantee here.
> > > > >
> > > > > If the null payload goes in to Kafka at time To + 2, then
> essentially
> > > you
> > > > > loose the second message produced by the producer at time To + 1.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > 

[GitHub] kafka pull request #1985: MINOR: A bunch of clean-ups related to usage of un...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1985

MINOR: A bunch of clean-ups related to usage of unused variables

There should be only 2 cases where these clean-ups have a functional impact:

1. Fix a bug in the newly introduced `fetchAllChildEntityConfigs` where we 
are concatenating
`Some(s)` instead of `s`.
2. Replaced repeated identical logs with a single log for the stale 
controller epoch case

The rest should just make the code easier to read and make it a bit less 
wasteful. I did this exercise because unused variables sometimes mask bugs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka remove-unused

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1985.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1985


commit 36f60b7ed572a357530696fcef36161784e4ab70
Author: Ismael Juma 
Date:   2016-10-07T01:50:05Z

Use `path` instead of `rootPath` when building entity type

commit 30a454590bdd7ff1c8931f579548dc5732bc32df
Author: Ismael Juma 
Date:   2016-10-07T01:55:47Z

Remove unused fields, variables and methods

commit 75e1c4d78c547b44cb32d99fa0efbf11745b144b
Author: Ismael Juma 
Date:   2016-10-07T01:56:19Z

Remove unnecessary initialisation

commit 7ad124832b69f4ca4f9e36ceb4eab53a764ff28a
Author: Ismael Juma 
Date:   2016-10-07T02:02:41Z

Use underscore instead of unused variables

commit 12cf7705f81e286cdfa6c1fbb7fed91a1f69acbc
Author: Ismael Juma 
Date:   2016-10-07T02:03:53Z

Minor exception handling clean-ups

commit d73e47f7ba907280ecc3f8096b3eca1d600bc66a
Author: Ismael Juma 
Date:   2016-10-07T02:08:16Z

Use `keys`, `values` and similar methods to avoid unused variables

commit 43b8709d8b17c5ff60f77f4b33999d11d690dfee
Author: Ismael Juma 
Date:   2016-10-07T02:09:03Z

Avoid unnecessary `toSet` and replace pattern matching by `isDefined`

commit e17d31467b3fbd4a7e9f2e870b37e69e987394d6
Author: Ismael Juma 
Date:   2016-10-07T02:10:24Z

Replace one log entry per partition with a single log entry for a stale 
controller epoch error

Since the same information was printed each time, it seems completely 
pointless to print it
multiple times.

commit 98324c448daf68e8392a163dfe618f712066ea35
Author: Ismael Juma 
Date:   2016-10-07T02:11:05Z

Controller clean-ups related to unused variables




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] 0.10.1.0 RC0

[Henry Cai]

Why is this feature in the release note?


   - [KAFKA-264 ] - Change
   the consumer side load balancing and distributed co-ordination to use a
   consumer co-ordinator

I thought this was already done in 2015.

On Thu, Oct 6, 2016 at 4:55 PM, Vahid S Hashemian  wrote:

> Jason,
>
> Thanks a lot for managing this release.
>
> I ran the quick start (Steps 2-8) with this release candidate on Ubuntu,
> Windows, and Mac and they mostly look great.
> These are some, hopefully, minor items and gaps I noticed with respect to
> the existing quick start documentation (and the updated quick start that
> leverages the new consumer).
> They may very well be carryovers from previous releases, or perhaps
> specific to my local environments.
> Hopefully others can confirm.
>
>
> Windows
>
> Since there are separate scripts on Windows platform, it probably would
> help if that is clarified in the quick start section. E.g. "On Windows
> platform replace `bin/` with `bin\windows\`". Or even have a separate
> quick start for Windows since a number of commands will be different on
> Windows.
> There is no `connect-standalone.sh` equivalent for Windows under
> bin\windows folder (Step 7).
> Step 8 is also not tailored for Windows terminals. I skipped this step.
> When I try to consume message using the new consumer (Step 5) I get an
> exception on the broker side. The old consumer works fine.
>
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> ReplicaManager.scala:740)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> ReplicaManager.scala:739)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.
> apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at
> kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(
> ReplicaManager.scala:685)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
>
> This issue seems to break the broker and I have to clear out the logs so I
> can bring the broker back up again.
>
>
> Ubuntu / Mac
>
> At Step 8, the output I'm seeing after going through the instructions in
> sequence is this (with unique words)
>
> all 1
> lead1
> to  1
> hello   1
> streams 2
> join1
> kafka   3
> summit  1
>
> which is different what I see in the documentation (with repeating words).
>
>
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: us...@kafka.apache.org, dev@kafka.apache.org, kafka-clients
> 
> Date:   10/04/2016 04:13 PM
> Subject:Re: [VOTE] 0.10.1.0 RC0
>
>
>
> One clarification: this is a minor release, not a major one.
>
> -Jason
>
> On Tue, Oct 4, 2016 at 4:01 PM, Jason Gustafson 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 0.10.1.0. This
> is
> > a major release that includes great new features including throttled
> > replication, secure quotas, time-based log searching, and queryable
> state
> > for Kafka Streams. A full list of the 

Build failed in Jenkins: kafka-trunk-jdk7 #1606

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4252; Fix purgatory metric name in documentation

--
[...truncated 2425 lines...]
kafka.api.AdminClientTest > testDescribeConsumerGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroup PASSED

kafka.api.AdminClientTest > testListGroups STARTED

kafka.api.AdminClientTest > testListGroups PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup PASSED

kafka.api.ProducerBounceTest > testBrokerFailure STARTED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[0] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[1] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[2] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[2] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[3] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[3] PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms STARTED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms PASSED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption PASSED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic STARTED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic PASSED

kafka.api.FetchRequestTest > testShuffle STARTED

kafka.api.FetchRequestTest > testShuffle PASSED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled 
STARTED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete STARTED

Re: [VOTE] 0.10.1.0 RC0

[Vahid S Hashemian]

Jason,

Thanks a lot for managing this release.

I ran the quick start (Steps 2-8) with this release candidate on Ubuntu, 
Windows, and Mac and they mostly look great.
These are some, hopefully, minor items and gaps I noticed with respect to 
the existing quick start documentation (and the updated quick start that 
leverages the new consumer).
They may very well be carryovers from previous releases, or perhaps 
specific to my local environments.
Hopefully others can confirm.


Windows

Since there are separate scripts on Windows platform, it probably would 
help if that is clarified in the quick start section. E.g. "On Windows 
platform replace `bin/` with `bin\windows\`". Or even have a separate 
quick start for Windows since a number of commands will be different on 
Windows.
There is no `connect-standalone.sh` equivalent for Windows under 
bin\windows folder (Step 7).
Step 8 is also not tailored for Windows terminals. I skipped this step.
When I try to consume message using the new consumer (Step 5) I get an 
exception on the broker side. The old consumer works fine.

java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(Unknown Source)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at 
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at 
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at 
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
at 
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at 
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at 
kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
at 
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
at 
kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at 
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
... 29 more

This issue seems to break the broker and I have to clear out the logs so I 
can bring the broker back up again.


Ubuntu / Mac

At Step 8, the output I'm seeing after going through the instructions in 
sequence is this (with unique words)

all 1
lead1
to  1
hello   1
streams 2
join1
kafka   3
summit  1

which is different what I see in the documentation (with repeating words).


--Vahid




From:   Jason Gustafson 
To: us...@kafka.apache.org, dev@kafka.apache.org, kafka-clients 

Date:   10/04/2016 04:13 PM
Subject:Re: [VOTE] 0.10.1.0 RC0



One clarification: this is a minor release, not a major one.

-Jason

On Tue, Oct 4, 2016 at 4:01 PM, Jason Gustafson  
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 0.10.1.0. This 
is
> a major release that includes great new features including throttled
> replication, secure quotas, time-based log searching, and queryable 
state
> for Kafka Streams. A full list of the content can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1. 
Since
> this is a major release, we will give people more time to try it out and
> give feedback.
>
> Release notes for the 0.10.1.0 release:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Oct 10, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> 

Build failed in Jenkins: kafka-trunk-jdk8 #951

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4252; Fix purgatory metric name in documentation

--
[...truncated 14069 lines...]
org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldThrowStreamsExceptionAfterMaxAttempts STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldThrowStreamsExceptionAfterMaxAttempts PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldRetryWhenTimeoutExceptionOccursOnSend STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
shouldRetryWhenTimeoutExceptionOccursOnSend PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithNewTasks STARTED

Re: [DISCUSS] KIP-82 - Add Record Headers

[Mayuresh Gharat]

+1 Nacho, Radai.

Ordering the Keys would help if we were gonna look at the headers linearly
but given the disadvantage that the client implementations have to know the
order of headers in order that the reading system in the pipeline doesn't
break, unordered list sounds better.

Thanks,

Mayuresh


On Thu, Oct 6, 2016 at 2:46 PM, Nacho Solis 
wrote:

> I'm also
>
> 1. no  (ordered keys)
> 2. yes (propose key space)
>
>
> 1. I don't think there is going to be much savings in ordering the keys.
> I'm assuming some parsing will happen either way. Ordering the keys would
> be useful if we were doing linear search on the headers, and even then, the
> performance difference would be small for any reasonable number of headers
> (even anything that fits in 1 meg).
>
> However, I think that it's likely that whoever is looking at the headers is
> going to want to search for plugins for any header in existence, as such,
> it's going to have to iterate over the whole header set. So, for every
> header, look up the plugin, not the other way around.  Even if we did it
> the other way around (for every plugin, search if there is a header
> present) we would expect only to have an algorithm that is O(n) and only
> iterate once over the list. We wouldn't need to iterate more than once.
>
> Given this, the code overhead of ordering the headers when something is
> inserted and such is a bigger pain than dealing with a potentially
> unordered list.
>
>
> 2. I like structure and to reduce the play space for potential keys.  This
> will allow us to do filter and know when we're testing. At the same time,
> we're reserving a lot of space for future usages. However, if there is no
> agreement on this I don't think it would be a blocker.  I just want to make
> sure we have some order and if possible contiguous ranges for similar
> usages.
>
> Nacho
>
>
> On Thu, Oct 6, 2016 at 2:26 PM, radai  wrote:
>
> > 1. tending towards no, but I dont have any strong opinions on header
> > ordering. it offers a potential speedup for header lookup in a serialized
> > blob (in wire format) but that goes away if the headers are fully
> > serialized/deserialized always. on the downside its an implementation
> > detail that 3rd party impls would need to worry about, and would be hard
> to
> > diagnose if they fail to. its also less friendly to high performance io
> > (think about appending headers to an existing blob in pass-through
> > components like mirror-maker vs write to the middle) - its still possible
> > though. however, the kafka code base is far from being iovec friendly
> > anyway.
> >
> > 2. yes.
> >
> >
> >
> >
> >
> > On Thu, Oct 6, 2016 at 8:58 AM, K Burstev  wrote:
> >
> > > @Mayuresh
> > >
> > > Yes exactly, it is a real nasty race issue.
> > >
> > > This is why I look forward to being able to trash our custom workaround
> > :)
> > >
> > > Kostya
> > >
> > >
> > > 06.10.2016, 02:36, "Mayuresh Gharat" :
> > > > @Kostya
> > > >
> > > > Regarding "To get around this we have an awful *cough* solution
> whereby
> > > we
> > > > have to send our message wrapper with the headers and null content,
> and
> > > > then we have an application that has to consume from all the
> compacted
> > > > topics and when it sees this message it produces back in a null
> payload
> > > > record to make the broker compact it out."
> > > >
> > > >  ---> This has a race condition, right?
> > > >
> > > > Suppose the producer produces a message with headers and null content
> > at
> > > > time To to Kafka.
> > > >
> > > > Then the producer, at time To + 1, sends another message with headers
> > and
> > > > actual content to Kafka.
> > > >
> > > > What we expect is that the application that is consuming and then
> > > producing
> > > > same message with null payload should happen at time To + 0.5, so
> that
> > > the
> > > > message at To + 1 is not deleted.
> > > >
> > > > But there is no guarantee here.
> > > >
> > > > If the null payload goes in to Kafka at time To + 2, then essentially
> > you
> > > > loose the second message produced by the producer at time To + 1.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy 
> > wrote:
> > > >
> > > >>  @Nacho
> > > >>
> > > >>  > > - Brokers can't see the headers (part of the "V" black box)>
> > > >>  >
> > > >>
> > > >>  > (Also, it would be nice if we had a way to access the headers
> from
> > > the
> > > >>  > > brokers, something that is not trivial at this time with the
> > > current
> > > >>  > broker
> > > >>  > > architecture).
> > > >>  >
> > > >>  >
> > > >>
> > > >>  I think this can be addressed with broker interceptors which we
> > > touched on
> > > >>  in KIP-42
> > > >>   > > >>  42%3A+Add+Producer+and+Consumer+Interceptors>
> > > >>  .
> > > >>
> > > >>  

[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553410#comment-15553410
 ] 

Jason Gustafson commented on KAFKA-4254:


[~jjkoshy] Yeah, good point.

> Questionable handling of unknown partitions in KafkaProducer
> 
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user 
> attempts to write to a partition which has just been created. This is caused 
> by the fact that the producer does not attempt to refetch topic metadata in 
> this case, which means that its check for partition validity is based on 
> stale metadata.
> If the topic for the partition did not already exist, it works fine because 
> the producer will block until it has metadata for the topic, so this case is 
> primarily hit when the number of partitions is dynamically increased. 
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the 
> partition doesn't exist, we refetch metadata and try again (timing out when 
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle 
> the error. Raising {{IllegalArgumentException}} is not helpful in practice 
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to 
> tell the user that the partition is invalid.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Nacho Solis]

I'm also

1. no  (ordered keys)
2. yes (propose key space)


1. I don't think there is going to be much savings in ordering the keys.
I'm assuming some parsing will happen either way. Ordering the keys would
be useful if we were doing linear search on the headers, and even then, the
performance difference would be small for any reasonable number of headers
(even anything that fits in 1 meg).

However, I think that it's likely that whoever is looking at the headers is
going to want to search for plugins for any header in existence, as such,
it's going to have to iterate over the whole header set. So, for every
header, look up the plugin, not the other way around.  Even if we did it
the other way around (for every plugin, search if there is a header
present) we would expect only to have an algorithm that is O(n) and only
iterate once over the list. We wouldn't need to iterate more than once.

Given this, the code overhead of ordering the headers when something is
inserted and such is a bigger pain than dealing with a potentially
unordered list.


2. I like structure and to reduce the play space for potential keys.  This
will allow us to do filter and know when we're testing. At the same time,
we're reserving a lot of space for future usages. However, if there is no
agreement on this I don't think it would be a blocker.  I just want to make
sure we have some order and if possible contiguous ranges for similar
usages.

Nacho


On Thu, Oct 6, 2016 at 2:26 PM, radai  wrote:

> 1. tending towards no, but I dont have any strong opinions on header
> ordering. it offers a potential speedup for header lookup in a serialized
> blob (in wire format) but that goes away if the headers are fully
> serialized/deserialized always. on the downside its an implementation
> detail that 3rd party impls would need to worry about, and would be hard to
> diagnose if they fail to. its also less friendly to high performance io
> (think about appending headers to an existing blob in pass-through
> components like mirror-maker vs write to the middle) - its still possible
> though. however, the kafka code base is far from being iovec friendly
> anyway.
>
> 2. yes.
>
>
>
>
>
> On Thu, Oct 6, 2016 at 8:58 AM, K Burstev  wrote:
>
> > @Mayuresh
> >
> > Yes exactly, it is a real nasty race issue.
> >
> > This is why I look forward to being able to trash our custom workaround
> :)
> >
> > Kostya
> >
> >
> > 06.10.2016, 02:36, "Mayuresh Gharat" :
> > > @Kostya
> > >
> > > Regarding "To get around this we have an awful *cough* solution whereby
> > we
> > > have to send our message wrapper with the headers and null content, and
> > > then we have an application that has to consume from all the compacted
> > > topics and when it sees this message it produces back in a null payload
> > > record to make the broker compact it out."
> > >
> > >  ---> This has a race condition, right?
> > >
> > > Suppose the producer produces a message with headers and null content
> at
> > > time To to Kafka.
> > >
> > > Then the producer, at time To + 1, sends another message with headers
> and
> > > actual content to Kafka.
> > >
> > > What we expect is that the application that is consuming and then
> > producing
> > > same message with null payload should happen at time To + 0.5, so that
> > the
> > > message at To + 1 is not deleted.
> > >
> > > But there is no guarantee here.
> > >
> > > If the null payload goes in to Kafka at time To + 2, then essentially
> you
> > > loose the second message produced by the producer at time To + 1.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy 
> wrote:
> > >
> > >>  @Nacho
> > >>
> > >>  > > - Brokers can't see the headers (part of the "V" black box)>
> > >>  >
> > >>
> > >>  > (Also, it would be nice if we had a way to access the headers from
> > the
> > >>  > > brokers, something that is not trivial at this time with the
> > current
> > >>  > broker
> > >>  > > architecture).
> > >>  >
> > >>  >
> > >>
> > >>  I think this can be addressed with broker interceptors which we
> > touched on
> > >>  in KIP-42
> > >>   > >>  42%3A+Add+Producer+and+Consumer+Interceptors>
> > >>  .
> > >>
> > >>  @Gwen
> > >>
> > >>  You are right that the wrapper thingy “works”, but there are some
> > drawbacks
> > >>  that Nacho and Radai have covered in detail that I can add a few more
> > >>  comments to.
> > >>
> > >>  At LinkedIn, we *get by* without the proposed Kafka record headers by
> > >>  dumping such metadata in one or two places:
> > >>
> > >> - Most of our applications use Avro, so for the most part we can
> > use an
> > >> explicit header field in the Avro schema. Topic owners are
> supposed
> > to
> > >> include this header in their schemas.
> > >> - A prefix to the payload that primarily contains the schema’s 

[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553228#comment-15553228
 ] 

Joel Koshy commented on KAFKA-4254:
---

It seems (2) would still help as there are use-cases which set {{max.block.ms}} 
to zero. So we can refresh metadata but also return a more specific exception.

> Questionable handling of unknown partitions in KafkaProducer
> 
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user 
> attempts to write to a partition which has just been created. This is caused 
> by the fact that the producer does not attempt to refetch topic metadata in 
> this case, which means that its check for partition validity is based on 
> stale metadata.
> If the topic for the partition did not already exist, it works fine because 
> the producer will block until it has metadata for the topic, so this case is 
> primarily hit when the number of partitions is dynamically increased. 
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the 
> partition doesn't exist, we refetch metadata and try again (timing out when 
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle 
> the error. Raising {{IllegalArgumentException}} is not helpful in practice 
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to 
> tell the user that the partition is invalid.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[radai]

1. tending towards no, but I dont have any strong opinions on header
ordering. it offers a potential speedup for header lookup in a serialized
blob (in wire format) but that goes away if the headers are fully
serialized/deserialized always. on the downside its an implementation
detail that 3rd party impls would need to worry about, and would be hard to
diagnose if they fail to. its also less friendly to high performance io
(think about appending headers to an existing blob in pass-through
components like mirror-maker vs write to the middle) - its still possible
though. however, the kafka code base is far from being iovec friendly
anyway.

2. yes.





On Thu, Oct 6, 2016 at 8:58 AM, K Burstev  wrote:

> @Mayuresh
>
> Yes exactly, it is a real nasty race issue.
>
> This is why I look forward to being able to trash our custom workaround :)
>
> Kostya
>
>
> 06.10.2016, 02:36, "Mayuresh Gharat" :
> > @Kostya
> >
> > Regarding "To get around this we have an awful *cough* solution whereby
> we
> > have to send our message wrapper with the headers and null content, and
> > then we have an application that has to consume from all the compacted
> > topics and when it sees this message it produces back in a null payload
> > record to make the broker compact it out."
> >
> >  ---> This has a race condition, right?
> >
> > Suppose the producer produces a message with headers and null content at
> > time To to Kafka.
> >
> > Then the producer, at time To + 1, sends another message with headers and
> > actual content to Kafka.
> >
> > What we expect is that the application that is consuming and then
> producing
> > same message with null payload should happen at time To + 0.5, so that
> the
> > message at To + 1 is not deleted.
> >
> > But there is no guarantee here.
> >
> > If the null payload goes in to Kafka at time To + 2, then essentially you
> > loose the second message produced by the producer at time To + 1.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy  wrote:
> >
> >>  @Nacho
> >>
> >>  > > - Brokers can't see the headers (part of the "V" black box)>
> >>  >
> >>
> >>  > (Also, it would be nice if we had a way to access the headers from
> the
> >>  > > brokers, something that is not trivial at this time with the
> current
> >>  > broker
> >>  > > architecture).
> >>  >
> >>  >
> >>
> >>  I think this can be addressed with broker interceptors which we
> touched on
> >>  in KIP-42
> >>   >>  42%3A+Add+Producer+and+Consumer+Interceptors>
> >>  .
> >>
> >>  @Gwen
> >>
> >>  You are right that the wrapper thingy “works”, but there are some
> drawbacks
> >>  that Nacho and Radai have covered in detail that I can add a few more
> >>  comments to.
> >>
> >>  At LinkedIn, we *get by* without the proposed Kafka record headers by
> >>  dumping such metadata in one or two places:
> >>
> >> - Most of our applications use Avro, so for the most part we can
> use an
> >> explicit header field in the Avro schema. Topic owners are supposed
> to
> >> include this header in their schemas.
> >> - A prefix to the payload that primarily contains the schema’s ID
> so we
> >> can deserialize the Avro. (We could use this for other use-cases as
> >>  well -
> >> i.e., move some of the above into this prefix blob.)
> >>
> >>  Dumping headers in the Avro schema pollutes the application’s data
> model
> >>  with data/service-infra-related fields that are unrelated to the
> underlying
> >>  topic; and forces the application to deserialize the entire blob
> whether or
> >>  not the headers are actually used. Conversely from an infrastructure
> >>  perspective, we would really like to not touch any application data.
> Our
> >>  infiltration of the application’s schema is a major reason why many at
> >>  LinkedIn sometimes assume that we (Kafka folks) are the shepherds for
> all
> >>  things Avro :)
> >>
> >>  Another drawback is that all this only works if everyone in the
> >>  organization is a good citizen and includes the header; and uses our
> >>  wrapper libraries - which is a good practice IMO - but may not always
> be
> >>  easy for open source projects that wish to directly use the Apache
> >>  producer/client. If instead we allow these headers to be inserted via
> >>  suitable interceptors outside the application payloads it would remove
> such
> >>  issues of separation in the data model and choice of clients.
> >>
> >>  Radai has enumerated a number of use-cases
> >>   >>  Case+for+Kafka+Headers>
> >>  and
> >>  I’m sure the broader community will have a lot more to add. The
> feature as
> >>  such would enable an ecosystem of plugins from different vendors that
> users
> >>  can mix and match in their data pipelines without requiring any
> specific
> >>  payload formats or client 

Re: Snazzy new look to our website

[Jason Gustafson]

Thanks Mickael and Jonathon for reporting the problem with the javdocs. The
links should be fixed now.

-Jason

On Thu, Oct 6, 2016 at 10:59 AM, Jonathan Bond 
wrote:

> Hi,
>
> I'm having a problem with the new website.  Whenever I try to follow a link
> to the Kafka javadoc, either from within the website or a google link - it
> always takes me to the top of the Kafka 0.10 documentation page.  I can't
> figure out how to get to the javadoc.
>
> Thanks, Jonathan
>
> On Tue, Oct 4, 2016 at 6:46 PM Jason Gustafson  wrote:
>
> > Huge improvement. Thanks Derrick and Gwen!
> >
> > On Tue, Oct 4, 2016 at 5:54 PM, Becket Qin  wrote:
> >
> > > Much fancier now :)
> > >
> > > On Tue, Oct 4, 2016 at 5:51 PM, Ali Akhtar 
> wrote:
> > >
> > > > Just noticed this on pulling up the documentation. Oh yeah! This new
> > look
> > > > is fantastic.
> > > >
> > > > On Wed, Oct 5, 2016 at 4:31 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com
> > > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Thank you for the much needed new design.
> > > > > At first glance, it looks great, and more professional.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > > From:   Gwen Shapira 
> > > > > To: dev@kafka.apache.org, Users 
> > > > > Cc: Derrick Or 
> > > > > Date:   10/04/2016 04:13 PM
> > > > > Subject:Snazzy new look to our website
> > > > >
> > > > >
> > > > >
> > > > > Hi Team Kafka,
> > > > >
> > > > > I just merged PR 20 to our website - which gives it a new (and IMO
> > > > > pretty snazzy) look and feel. Thanks to Derrick Or for contributing
> > > > > the update.
> > > > >
> > > > > I had to do a hard-refresh (shift-f5 on my mac) to get the new look
> > to
> > > > > load properly - so if stuff looks off, try it.
> > > > >
> > > > > Comments and contributions to the site are welcome.
> > > > >
> > > > > Gwen
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-0.10.1-jdk7 #50

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4252; Fix purgatory metric name in documentation

--
[...truncated 3797 lines...]

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover STARTED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower STARTED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic STARTED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion STARTED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic STARTED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition STARTED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenRacks 
STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenRacks 
PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenReplicas 
STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAwareWithUnevenReplicas 
PASSED

kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned 
STARTED

kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlreadyAssigned PASSED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware STARTED

kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware PASSED

kafka.admin.AdminRackAwareTest > testRackAwareExpansion STARTED

kafka.admin.AdminRackAwareTest > testRackAwareExpansion PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6Partitions STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6Partitions PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6PartitionsAnd3Brokers STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWith2ReplicasRackAwareWith6PartitionsAnd3Brokers PASSED

kafka.admin.AdminRackAwareTest > 
testGetRackAlternatedBrokerListAndAssignReplicasToBrokers STARTED

kafka.admin.AdminRackAwareTest > 
testGetRackAlternatedBrokerListAndAssignReplicasToBrokers PASSED

kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks STARTED

kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks PASSED

kafka.admin.AdminRackAwareTest > testSingleRack STARTED

kafka.admin.AdminRackAwareTest > testSingleRack PASSED

kafka.admin.AdminRackAwareTest > 
testAssignmentWithRackAwareWithRandomStartIndex STARTED

kafka.admin.AdminRackAwareTest > 
testAssignmentWithRackAwareWithRandomStartIndex PASSED

kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment STARTED

kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment PASSED

kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks STARTED

kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup STARTED

kafka.admin.DeleteConsumerGroupTest > 
testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics 
STARTED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics 
PASSED

kafka.admin.DeleteConsumerGroupTest > 
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK STARTED

kafka.admin.DeleteConsumerGroupTest > 

[GitHub] kafka-site pull request #22: fix issue with rewrite rules blocking access to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/22


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[Cody Koeninger (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553085#comment-15553085
 ] 

Cody Koeninger commented on KAFKA-3370:
---

Throwing an exception should be the default IMHO.  Silent data loss mid-stream 
isn't cool.

> Add options to auto.offset.reset to reset offsets upon initialization only
> --
>
> Key: KAFKA-3370
> URL: https://issues.apache.org/jira/browse/KAFKA-3370
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> Currently "auto.offset.reset" is applied in the following two cases:
> 1) upon starting the consumer for the first time (hence no committed offsets 
> before);
> 2) upon fetching offsets out-of-range.
> For scenarios where case 2) needs to be avoid (i.e. people need to be 
> notified upon offsets out-of-range rather than silently offset reset), 
> "auto.offset.reset" need to be set to "none". However for case 1) setting 
> "auto.offset.reset" to "none" will cause NoOffsetForPartitionException upon 
> polling. And in this case, seekToBeginning/seekToEnd is mistakenly applied 
> trying to set the offset at initialization, which are actually designed for 
> during the life time of the consumer (in rebalance callback, for example).
> The fix proposal is to add two more options to "auto.offset.reset", 
> "earliest-on-start", and "latest-on-start", whose semantics are "earliest" 
> and "latest" for case 1) only, and "none" for case 2).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #950

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Clarify 0.10.1.0 upgrade docs

[wangguoz] KAFKA-4253: Fix Kafka Stream thread shutting down process ordering

--
[...truncated 2403 lines...]
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:574)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:381)
at hudson.scm.SCM.poll(SCM.java:398)
at hudson.model.AbstractProject._poll(AbstractProject.java:1446)
at hudson.model.AbstractProject.poll(AbstractProject.java:1349)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:528)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:557)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

kafka.api.AdminClientTest > testDescribeGroup PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroup PASSED

kafka.api.AdminClientTest > testListGroups STARTED

kafka.api.AdminClientTest > testListGroups PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup PASSED

kafka.api.ProducerBounceTest > testBrokerFailure STARTED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[0] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[1] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[2] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[2] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[3] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[3] PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures STARTED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > 

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[Marcin Kuthan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553047#comment-15553047
 ] 

Marcin Kuthan commented on KAFKA-3370:
--

As I understand we need two separate properties:

auto.reset.offset: for new topic/partition scenario with two options (keep 
existing property name/value for backward compatiblity):
* earliest
* latest

auto.reset.offset.existing: for existing topic/partition scenario also with two 
options:
* nearest - move to earliest if offset too low, move to latest if offset too 
high
* none - throw an exception

I'm not sure which one should be treated as default, perhaps none because it 
seems to be safer.

What do you think?

> Add options to auto.offset.reset to reset offsets upon initialization only
> --
>
> Key: KAFKA-3370
> URL: https://issues.apache.org/jira/browse/KAFKA-3370
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> Currently "auto.offset.reset" is applied in the following two cases:
> 1) upon starting the consumer for the first time (hence no committed offsets 
> before);
> 2) upon fetching offsets out-of-range.
> For scenarios where case 2) needs to be avoid (i.e. people need to be 
> notified upon offsets out-of-range rather than silently offset reset), 
> "auto.offset.reset" need to be set to "none". However for case 1) setting 
> "auto.offset.reset" to "none" will cause NoOffsetForPartitionException upon 
> polling. And in this case, seekToBeginning/seekToEnd is mistakenly applied 
> trying to set the offset at initialization, which are actually designed for 
> during the life time of the consumer (in rebalance callback, for example).
> The fix proposal is to add two more options to "auto.offset.reset", 
> "earliest-on-start", and "latest-on-start", whose semantics are "earliest" 
> and "latest" for case 1) only, and "none" for case 2).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.1-jdk7 #49

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Clarify 0.10.1.0 upgrade docs

[wangguoz] KAFKA-4253: Fix Kafka Stream thread shutting down process ordering

--
[...truncated 7449 lines...]

kafka.server.DynamicConfigTest > shouldFailLeaderConfigsWithInvalidValues PASSED

kafka.server.DynamicConfigTest > shouldFailWhenChangingClientIdUnknownConfig 
STARTED

kafka.server.DynamicConfigTest > shouldFailWhenChangingClientIdUnknownConfig 
PASSED

kafka.server.DynamicConfigTest > shouldFailWhenChangingBrokerUnknownConfig 
STARTED

kafka.server.DynamicConfigTest > shouldFailWhenChangingBrokerUnknownConfig 
PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.LeaderElectionTest > testLeaderElectionWithStaleControllerEpoch 
STARTED

kafka.server.LeaderElectionTest > testLeaderElectionWithStaleControllerEpoch 
PASSED

kafka.server.LeaderElectionTest > testLeaderElectionAndEpoch STARTED

kafka.server.LeaderElectionTest > testLeaderElectionAndEpoch PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.DynamicConfigChangeTest > testProcessNotification STARTED

kafka.server.DynamicConfigChangeTest > testProcessNotification PASSED

kafka.server.DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties STARTED

kafka.server.DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties PASSED

kafka.server.DynamicConfigChangeTest > testDefaultClientIdQuotaConfigChange 
STARTED

kafka.server.DynamicConfigChangeTest > testDefaultClientIdQuotaConfigChange 
PASSED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange STARTED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange PASSED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
STARTED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
PASSED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues STARTED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues PASSED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > 

[GitHub] kafka-site pull request #23: Grammar updated

[muehlburger]

GitHub user muehlburger opened a pull request:

https://github.com/apache/kafka-site/pull/23

Grammar updated

Just a little grammar update.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/muehlburger/kafka-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/23.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23


commit a7344f65c7b23ec3e455169711a430a76468cd4a
Author: Herbert Mühlburger 
Date:   2016-10-06T19:16:44Z

Grammar updated

Just a little grammar update.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1980: KAFKA-4252: Fix metric name in documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1980


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4252) Missing ProducerRequestPurgatory

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4252:
---
   Resolution: Fixed
Fix Version/s: 0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1980
[https://github.com/apache/kafka/pull/1980]

> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Fix For: 0.10.1.0
>
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4252) Missing ProducerRequestPurgatory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552728#comment-15552728
 ] 

ASF GitHub Bot commented on KAFKA-4252:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1980


> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Fix For: 0.10.1.0
>
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer

[Konstantine Karantasis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis reassigned KAFKA-4254:
-

Assignee: Konstantine Karantasis  (was: Jason Gustafson)

> Questionable handling of unknown partitions in KafkaProducer
> 
>
> Key: KAFKA-4254
> URL: https://issues.apache.org/jira/browse/KAFKA-4254
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Jason Gustafson
>Assignee: Konstantine Karantasis
> Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user 
> attempts to write to a partition which has just been created. This is caused 
> by the fact that the producer does not attempt to refetch topic metadata in 
> this case, which means that its check for partition validity is based on 
> stale metadata.
> If the topic for the partition did not already exist, it works fine because 
> the producer will block until it has metadata for the topic, so this case is 
> primarily hit when the number of partitions is dynamically increased. 
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the 
> partition doesn't exist, we refetch metadata and try again (timing out when 
> max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle 
> the error. Raising {{IllegalArgumentException}} is not helpful in practice 
> because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to 
> tell the user that the partition is invalid.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Snazzy new look to our website

[Jonathan Bond]

Hi,

I'm having a problem with the new website.  Whenever I try to follow a link
to the Kafka javadoc, either from within the website or a google link - it
always takes me to the top of the Kafka 0.10 documentation page.  I can't
figure out how to get to the javadoc.

Thanks, Jonathan

On Tue, Oct 4, 2016 at 6:46 PM Jason Gustafson  wrote:

> Huge improvement. Thanks Derrick and Gwen!
>
> On Tue, Oct 4, 2016 at 5:54 PM, Becket Qin  wrote:
>
> > Much fancier now :)
> >
> > On Tue, Oct 4, 2016 at 5:51 PM, Ali Akhtar  wrote:
> >
> > > Just noticed this on pulling up the documentation. Oh yeah! This new
> look
> > > is fantastic.
> > >
> > > On Wed, Oct 5, 2016 at 4:31 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com
> > > > wrote:
> > >
> > > > +1
> > > >
> > > > Thank you for the much needed new design.
> > > > At first glance, it looks great, and more professional.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From:   Gwen Shapira 
> > > > To: dev@kafka.apache.org, Users 
> > > > Cc: Derrick Or 
> > > > Date:   10/04/2016 04:13 PM
> > > > Subject:Snazzy new look to our website
> > > >
> > > >
> > > >
> > > > Hi Team Kafka,
> > > >
> > > > I just merged PR 20 to our website - which gives it a new (and IMO
> > > > pretty snazzy) look and feel. Thanks to Derrick Or for contributing
> > > > the update.
> > > >
> > > > I had to do a hard-refresh (shift-f5 on my mac) to get the new look
> to
> > > > load properly - so if stuff looks off, try it.
> > > >
> > > > Comments and contributions to the site are welcome.
> > > >
> > > > Gwen
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-82 - Add Record Headers

[K Burstev]

@Mayuresh

Yes exactly, it is a real nasty race issue.

This is why I look forward to being able to trash our custom workaround :)

Kostya


06.10.2016, 02:36, "Mayuresh Gharat" :
> @Kostya
>
> Regarding "To get around this we have an awful *cough* solution whereby we
> have to send our message wrapper with the headers and null content, and
> then we have an application that has to consume from all the compacted
> topics and when it sees this message it produces back in a null payload
> record to make the broker compact it out."
>
>  ---> This has a race condition, right?
>
> Suppose the producer produces a message with headers and null content at
> time To to Kafka.
>
> Then the producer, at time To + 1, sends another message with headers and
> actual content to Kafka.
>
> What we expect is that the application that is consuming and then producing
> same message with null payload should happen at time To + 0.5, so that the
> message at To + 1 is not deleted.
>
> But there is no guarantee here.
>
> If the null payload goes in to Kafka at time To + 2, then essentially you
> loose the second message produced by the producer at time To + 1.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy  wrote:
>
>>  @Nacho
>>
>>  > > - Brokers can't see the headers (part of the "V" black box)>
>>  >
>>
>>  > (Also, it would be nice if we had a way to access the headers from the
>>  > > brokers, something that is not trivial at this time with the current
>>  > broker
>>  > > architecture).
>>  >
>>  >
>>
>>  I think this can be addressed with broker interceptors which we touched on
>>  in KIP-42
>>  >  42%3A+Add+Producer+and+Consumer+Interceptors>
>>  .
>>
>>  @Gwen
>>
>>  You are right that the wrapper thingy “works”, but there are some drawbacks
>>  that Nacho and Radai have covered in detail that I can add a few more
>>  comments to.
>>
>>  At LinkedIn, we *get by* without the proposed Kafka record headers by
>>  dumping such metadata in one or two places:
>>
>> - Most of our applications use Avro, so for the most part we can use an
>> explicit header field in the Avro schema. Topic owners are supposed to
>> include this header in their schemas.
>> - A prefix to the payload that primarily contains the schema’s ID so we
>> can deserialize the Avro. (We could use this for other use-cases as
>>  well -
>> i.e., move some of the above into this prefix blob.)
>>
>>  Dumping headers in the Avro schema pollutes the application’s data model
>>  with data/service-infra-related fields that are unrelated to the underlying
>>  topic; and forces the application to deserialize the entire blob whether or
>>  not the headers are actually used. Conversely from an infrastructure
>>  perspective, we would really like to not touch any application data. Our
>>  infiltration of the application’s schema is a major reason why many at
>>  LinkedIn sometimes assume that we (Kafka folks) are the shepherds for all
>>  things Avro :)
>>
>>  Another drawback is that all this only works if everyone in the
>>  organization is a good citizen and includes the header; and uses our
>>  wrapper libraries - which is a good practice IMO - but may not always be
>>  easy for open source projects that wish to directly use the Apache
>>  producer/client. If instead we allow these headers to be inserted via
>>  suitable interceptors outside the application payloads it would remove such
>>  issues of separation in the data model and choice of clients.
>>
>>  Radai has enumerated a number of use-cases
>>  >  Case+for+Kafka+Headers>
>>  and
>>  I’m sure the broader community will have a lot more to add. The feature as
>>  such would enable an ecosystem of plugins from different vendors that users
>>  can mix and match in their data pipelines without requiring any specific
>>  payload formats or client libraries.
>>
>>  Thanks,
>>
>>  Joel
>>
>>  > >
>>  > >
>>  > > On Wed, Oct 5, 2016 at 2:20 PM, Gwen Shapira 
>>  wrote:
>>  > >
>>  > > > Since LinkedIn has some kind of wrapper thingy that adds the headers,
>>  > > > where they could have added them to Apache Kafka - I'm very curious
>>  to
>>  > > > hear what drove that decision and the pros/cons of managing the
>>  > > > headers outside Kafka itself.
>>  > > >
>>  >
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125

[jira] [Commented] (KAFKA-4264) kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh

[Alex Schmitz (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552637#comment-15552637
 ] 

Alex Schmitz commented on KAFKA-4264:
-

Pull request with fix here: https://github.com/apache/kafka/pull/1984

> kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh
> --
>
> Key: KAFKA-4264
> URL: https://issues.apache.org/jira/browse/KAFKA-4264
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.10.0.1
> Environment: Tested in Debian Jessy
>Reporter: Alex Schmitz
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> kafka-server-stop.sh greps for the process ID to kill with the following: 
> bq. PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk 
> '{print $1}')
> However, if Kafka is launched via the kafka-server-start.sh script, the 
> process doesn't include kafka.Kafka, the grep fails to find the process, and 
> it returns the failure message, No Kafka server to stop. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4264) kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552632#comment-15552632
 ] 

ASF GitHub Bot commented on KAFKA-4264:
---

GitHub user davemcphee opened a pull request:

https://github.com/apache/kafka/pull/1984

mods grep target to find kafka process, fixes KAFKA-4264

Fixes issue described here: https://issues.apache.org/jira/browse/KAFKA-4264

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davemcphee/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1984


commit 6ad83c5ad899f4ea7bda9a70fd350200eb2b
Author: Alex Schmitz 
Date:   2016-10-06T17:41:16Z

mods grep target to find kafka process, fixes KAFKA-4264




> kafka-server-stop.sh fails is Kafka launched via kafka-server-start.sh
> --
>
> Key: KAFKA-4264
> URL: https://issues.apache.org/jira/browse/KAFKA-4264
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.10.0.1
> Environment: Tested in Debian Jessy
>Reporter: Alex Schmitz
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> kafka-server-stop.sh greps for the process ID to kill with the following: 
> bq. PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk 
> '{print $1}')
> However, if Kafka is launched via the kafka-server-start.sh script, the 
> process doesn't include kafka.Kafka, the grep fails to find the process, and 
> it returns the failure message, No Kafka server to stop. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1984: mods grep target to find kafka process, fixes KAFK...

[davemcphee]

GitHub user davemcphee opened a pull request:

https://github.com/apache/kafka/pull/1984

mods grep target to find kafka process, fixes KAFKA-4264

Fixes issue described here: https://issues.apache.org/jira/browse/KAFKA-4264

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davemcphee/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1984


commit 6ad83c5ad899f4ea7bda9a70fd350200eb2b
Author: Alex Schmitz 
Date:   2016-10-06T17:41:16Z

mods grep target to find kafka process, fixes KAFKA-4264




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3993) Console producer drops data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552622#comment-15552622
 ] 

ASF GitHub Bot commented on KAFKA-3993:
---

Github user theduderog closed the pull request at:

https://github.com/apache/kafka/pull/1667


> Console producer drops data
> ---
>
> Key: KAFKA-3993
> URL: https://issues.apache.org/jira/browse/KAFKA-3993
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>
> The console producer drops data when if the process exits too quickly.  I 
> suspect that the shutdown hook does not call close() or something goes wrong 
> during that close().
> Here's a simple to illustrate the issue:
> {noformat}
> export BOOTSTRAP_SERVERS=localhost:9092
> export TOPIC=bar
> export MESSAGES=1
> ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 1 
> --replication-factor 1 --topic "$TOPIC" \
> && echo "acks=all" > /tmp/producer.config \
> && echo "linger.ms=0" >> /tmp/producer.config \
> && seq "$MESSAGES" | ./bin/kafka-console-producer.sh --broker-list 
> "$BOOTSTRAP_SERVERS" --topic "$TOPIC" --producer-config /tmp/producer.config \
> && ./bin/kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP_SERVERS" 
> --new-consumer --from-beginning --max-messages "${MESSAGES}" --topic "$TOPIC"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1667: KAFKA-3993 Console Producer Drops Data

[theduderog]

Github user theduderog closed the pull request at:

https://github.com/apache/kafka/pull/1667


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4263) QueryableStateIntegrationTest.concurrentAccess is failing occasionally in jenkins builds

[Damian Guy (JIRA)]

Damian Guy created KAFKA-4263:
-

 Summary: QueryableStateIntegrationTest.concurrentAccess is failing 
occasionally in jenkins builds
 Key: KAFKA-4263
 URL: https://issues.apache.org/jira/browse/KAFKA-4263
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.1.0
Reporter: Damian Guy
Assignee: Damian Guy
 Fix For: 0.10.1.1


We are seeing occasional failures of this test in jenkins, however it isn't 
failing when running locally (confirmed by multiple people). Needs 
investingating



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

Hi Manikumar,

I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.

-Jay

On Sat, Oct 1, 2016 at 5:38 AM, Manikumar  wrote:

> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official"  REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools.  Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar 
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available.  But we would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

[jira] [Commented] (KAFKA-4253) Fix Kafka Stream thread shutting down process ordering

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552447#comment-15552447
 ] 

ASF GitHub Bot commented on KAFKA-4253:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1970


> Fix Kafka Stream thread shutting down process ordering
> --
>
> Key: KAFKA-4253
> URL: https://issues.apache.org/jira/browse/KAFKA-4253
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Guozhang Wang
>Assignee: Damian Guy
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> Currently we close the stream thread in the following way:
> 0. Commit all tasks.
> 1. Close producer.
> 2. Close consumer.
> 3. Close restore consumer.
> 4. For each task, close its topology processors one-by-one following the 
> topology order by calling {{processor.close()}}.
> 5. For each task, close its state manager as well as flushing and closing all 
> its associated state stores.
> We choose to close the producer / consumer clients before shutting down the 
> tasks because we need to make sure all sent records has been acked so that we 
> have the right log-end-offset when closing the store and checkpointing the 
> offset of the changelog. However there is also an issue with this ordering, 
> in which users choose to write more records in their {{processor.close()}} 
> calls, this will cause RTE since the producers has already been closed, and 
> no changelog records will be able to write.
> Thinking about this issue, a more appropriate ordering will be:
> 1. For each task, close their topology processors following the topology 
> order by calling {{processor.close()}}.
> 2. For each task, commit its state by calling {{task.commit()}}. At this time 
> all sent records should be acked since {{producer.flush()}} is called.
> 3. For each task, close their {{ProcessorStateManager}}.
> 4. Close all embedded clients, i.e. producer / consumer / restore consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1970: KAFKA-4253: Fix Kafka Stream thread shutting down ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1970


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4253) Fix Kafka Stream thread shutting down process ordering

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4253:
-
   Resolution: Fixed
Fix Version/s: 0.10.1.1
   0.10.2.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1970
[https://github.com/apache/kafka/pull/1970]

> Fix Kafka Stream thread shutting down process ordering
> --
>
> Key: KAFKA-4253
> URL: https://issues.apache.org/jira/browse/KAFKA-4253
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Guozhang Wang
>Assignee: Damian Guy
> Fix For: 0.10.2.0, 0.10.1.1
>
>
> Currently we close the stream thread in the following way:
> 0. Commit all tasks.
> 1. Close producer.
> 2. Close consumer.
> 3. Close restore consumer.
> 4. For each task, close its topology processors one-by-one following the 
> topology order by calling {{processor.close()}}.
> 5. For each task, close its state manager as well as flushing and closing all 
> its associated state stores.
> We choose to close the producer / consumer clients before shutting down the 
> tasks because we need to make sure all sent records has been acked so that we 
> have the right log-end-offset when closing the store and checkpointing the 
> offset of the changelog. However there is also an issue with this ordering, 
> in which users choose to write more records in their {{processor.close()}} 
> calls, this will cause RTE since the producers has already been closed, and 
> no changelog records will be able to write.
> Thinking about this issue, a more appropriate ordering will be:
> 1. For each task, close their topology processors following the topology 
> order by calling {{processor.close()}}.
> 2. For each task, commit its state by calling {{task.commit()}}. At this time 
> all sent records should be acked since {{producer.flush()}} is called.
> 3. For each task, close their {{ProcessorStateManager}}.
> 4. Close all embedded clients, i.e. producer / consumer / restore consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1971: MINOR: Clarify 0.10.1.0 upgrade docs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1971


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-83 - Allow multiple SASL PLAIN authenticated Java clients in a single JVM process

[Edoardo Comar]

HI
after Rajini's excellent KIP-85 I am no longer sure KIP-83 needs to exist 
as is.

Building on top of KIP-85, all it takes to achieve KIP-83 goal is to 
change the caching of LoginManager on the client side.
For example using the new sasl.jaas.config value as the key.

This should solve both JIRAs 
https://issues.apache.org/jira/browse/KAFKA-4180
https://issues.apache.org/jira/browse/KAFKA-3302

Edo
--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU



From:   Edoardo Comar/UK/IBM@IBMGB
To: dev@kafka.apache.org
Date:   28/09/2016 10:55
Subject:Re: [DISCUSS] KIP-83 - Allow multiple SASL PLAIN 
authenticated Java clients in a single JVM process



Thanks Rajini and Harsha
I'll update the KIP 
--
Edoardo Comar
MQ Cloud Technologies
eco...@uk.ibm.com
+44 (0)1962 81 5576 
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 

3AU



From:   Rajini Sivaram 
To: dev@kafka.apache.org
Date:   28/09/2016 10:40
Subject:Re: [DISCUSS] KIP-83 - Allow multiple SASL PLAIN 
authenticated Java clients in a single JVM process



Edo,

I was going to write a KIP for this next week :-) I was thinking along the
same lines as Harsha, i.e., enable multiple LoginManagers to co-exist in a
JVM. The multi-user login module in MessageHub made sense at the time to
overcome the limitation in Kafka, without changing Kafka itself. But for
the KIP, it would be better to have a solution that supports multiple 
users
for any SASL mechanism.


On Wed, Sep 28, 2016 at 5:57 AM, Harsha Chintalapani 
wrote:

> Edorado,
> Thanks for the KIP. As pointed out in the JIRA can you make
> sure this is not just a specific change for SASL plain but make changes 
in
> general to LoginManager such that its not a singleton.
>
> Thanks,
> Harsha
>
> On Tue, Sep 27, 2016 at 10:15 AM Edoardo Comar  
wrote:
>
> > Hi,
> > I had a go at a KIP that addresses this JIRA
> > https://issues.apache.org/jira/browse/KAFKA-4180
> > "Shared authentification with multiple actives Kafka 
producers/consumers"
> >
> > which is a limitation of the current Java client that we (IBM 
MessageHub)
> > get asked quite often lately.
> >
> > We will have a go at a PR soon, just as a proof of concept, but as it
> > introduces new public interfaces it needs a KIP.
> >
> > I'll welcome your input.
> >
> > Edo
> > --
> > Edoardo Comar
> > MQ Cloud Technologies
> > eco...@uk.ibm.com
> > +44 (0)1962 81 5576
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> > IBM United Kingdom Limited Registered in England and Wales with number
> > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > 3AU
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with 
number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>



-- 
Regards,

Rajini



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[GitHub] kafka pull request #1983: client fails trying to bind to jmx port already in...

[edi-bice]

GitHub user edi-bice opened a pull request:

https://github.com/apache/kafka/pull/1983

client fails trying to bind to jmx port already in use by server

Enabling JMX_PORT causes Kafka clients to attempt to bind to a port already 
bound by server. Modified so as to distinguish if being called to launch server 
or client and choose accordingly whether to add JMX port option or not.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/edi-bice/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1983.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1983


commit 2c5d40e946bcc149b1a9b2c01eced4ae47a734c5
Author: Edi Bice 
Date:   2016-10-06T15:33:04Z

client fails trying to bind to jmx port already in use by server

Enabling JMX_PORT causes Kafka clients to attempt to bind to a port already 
bound by server. Modified so as to distinguish if being called to launch server 
or client and choose accordingly whether to add JMX port option or not.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ofir Manor]

I personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out.

However, there are two valid concerns here that could be addressed, around
community and accessibility:
>> What we are worried about is a project
>> that's not maintained in a community. So the process of accepting patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
and
>> hence there is no security support either.

This might be easy to fix. Maybe Confluent / kafka-rest community can
clarify that - what is their contribution policy, dev style, roadmap etc.
If they want, they can make an effort to encourage participation from
people outside Confluent (easily accept contributions, invite external
commiters or have open dev process similar to Apache Kafka etc), as there
is definitely seems to be some interest on the list. That might clear the
community concern and help kafka-rest project (but that is a calculation
Confluent will have to make).

The other, independent, concern is that REST is something that is expected
to be available out of the box with Kafka. I personally don't feel strongly
about it (better use proper, efficient APIs from day one), though it is
definitely way smaller than adding a stream processing engine to the
project :)
Again,the kafka-rest "community" could take steps to make it even easier to
install, configure and run kafka-rest for new users on vanilla Apache Kafka
(outside the Confluent platform), if they wish that (or welcome
contributions to that end), but that is up to them.
Finally, if after the above steps were taken there would still a strong
desire to include a great rest gateway with Apache Kafka, I assume the
community could hypothetically fork the existing kafka-rest into an Apache
Kafka subproject and maintain it "within Apache" instead of implementing it
from scratch (though I'm not a lawyer etc) - but I cannot imagine it happen
without Confluent blessing, and I think that is likely much less optimal
(pulling in other Confluent / Apache licensed dependencies) than having a
separate external community around kafka-rest.


Just my two cents,


Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani 
wrote:

> Neha & Jay,
>  We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community.  Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as  NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
>Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
>Kafka users, and maybe 10% of those are the ones who will use the REST
>proxy. The remaining 50% are non-java client users (C, python, go, node
>etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
>  Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
>to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub 

[GitHub] kafka pull request #1982: KAFKA-4262: Increase data volume in replication te...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1982

KAFKA-4262: Increase data volume in replication test

To prevent test from completing without throttling before config change 
takes effect, produce more messages in the test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4262

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1982.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1982


commit c9259001ec0983fcbd6d903439363dc8bbe0312e
Author: Rajini Sivaram 
Date:   2016-10-06T14:41:04Z

KAFKA-4262: Increase data volume in replication test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4262) Intermittent unit test failure ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552112#comment-15552112
 ] 

ASF GitHub Bot commented on KAFKA-4262:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1982

KAFKA-4262: Increase data volume in replication test

To prevent test from completing without throttling before config change 
takes effect, produce more messages in the test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4262

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1982.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1982


commit c9259001ec0983fcbd6d903439363dc8bbe0312e
Author: Rajini Sivaram 
Date:   2016-10-06T14:41:04Z

KAFKA-4262: Increase data volume in replication test




> Intermittent unit test failure 
> ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment
> ---
>
> Key: KAFKA-4262
> URL: https://issues.apache.org/jira/browse/KAFKA-4262
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> Unit test seen in PR build where PR did not contain any code changes: 
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6086/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldExecuteThrottledReassignment/
> {quote}
> java.lang.AssertionError: Expected replication to be > 4500.0 but was 210
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> kafka.admin.ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment(ReassignPartitionsClusterTest.scala:141)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ...
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4262) Intermittent unit test failure ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-4262:
-

 Summary: Intermittent unit test failure 
ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment
 Key: KAFKA-4262
 URL: https://issues.apache.org/jira/browse/KAFKA-4262
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


Unit test seen in PR build where PR did not contain any code changes: 
https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6086/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldExecuteThrottledReassignment/

{quote}
java.lang.AssertionError: Expected replication to be > 4500.0 but was 210
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
kafka.admin.ReassignPartitionsClusterTest.shouldExecuteThrottledReassignment(ReassignPartitionsClusterTest.scala:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3544) Missing topics on startup

[Mykola Polonskyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552065#comment-15552065
 ] 

Mykola Polonskyi commented on KAFKA-3544:
-

[~guozhang] greetings!
My comment a bit late but I catch error with similar stack-trace (like Greg), 
with using of KGroupedTable#aggregation and the intermediate topic.
kafka streams 0.10.0.1
{code:kotlin}
private val streamId = "STREAM_USER_WITH_SKICARD"
connectionProperties.put(StreamsConfig.APPLICATION_ID_CONFIG, streamId)
val skicardByOwnerIdTopicName = "skicards_by_owner_id"
userTable.join(
skicardsTable.groupBy {
key, value -> KeyValue(value.skicardInfo.ownerId, value.skicardInfo)
}.aggregate(
{ mutableSetOf() }, // accumulator definition
{ ownerIdString, skicardInfoObject, accumulator -> 
accumulator.put(skicardInfo) },
{ ownerIdString, skicardInfoObject, accumulator -> 
accumulator }, //XXX fix this one
skicardByOwnerIdSerde,
skicardByOwnerIdTopicName
),
{ userCreatedOrUpdated, skicardInfoSet -> 
UserWithSkicardsCreatedOrUpdated(userCreatedOrUpdated.user, skicardInfoSet) }
).to(
Serdes.StringSerde(),
userWithSkicardsSerde,
USER_WITH_SKICARDS_TOPIC.name
)
{code}

And inside kafka got appeared topic 
"STREAM_USER_WITH_SKICARD-skicards_by_owner_id-repartition"

And then incide the debug I see that code try to retrieve meta-info 
(StreamPartitionAssignor.java:446) for topic skicards_by_owner_id-repartition 
instead "STREAM_USER_WITH_SKICARD-skicards_by_owner_id-repartition".

Is that bug that you spoke about? 
p.s. I`m not sure should I report it as new ticket that`s why provide the 
additional comment here. 

> Missing topics on startup
> -
>
> Key: KAFKA-3544
> URL: https://issues.apache.org/jira/browse/KAFKA-3544
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
>  Labels: semantics
>
> When running a relatively complex job with multiple tasks and state stores, 
> on the first run I get errors due to some of the intermediate topics not 
> existing. Subsequent runs work OK. My assumption is streams may be creating 
> topics lazily, so if downstream tasks are initializing before their parents 
> have had a chance to create their necessary topics then the children will 
> attempt to start consuming from topics that do not exist yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Issue while closing consumer

[Maria Khan]

hi ,
I have an issue in kafka..I am using inbound channel adapter for spring
kafka in kafka consumer consumer code.when i undeploy my consumer
application ..i tried to close the adapter but it is still consuming
messages even if it is undeployed .it does not stop consuming messages

[jira] [Commented] (KAFKA-4260) Improve documentation of configuration listeners=PLAINTEXT://0.0.0.0:9092

[David Watzke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551925#comment-15551925
 ] 

David Watzke commented on KAFKA-4260:
-

Also quite disturbing was the fact that there was no relevant error message in 
logs that would tell us what the hell was happening - that should be addressed 
as well. Thanks.

> Improve documentation of configuration listeners=PLAINTEXT://0.0.0.0:9092
> -
>
> Key: KAFKA-4260
> URL: https://issues.apache.org/jira/browse/KAFKA-4260
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 0.10.0.1
>Reporter: Michal Turek
>Priority: Minor
>
> We have just updated our testing Kafka cluster to 0.10 and we were facing one 
> issue with migration of legacy 0.8 configuration to
> {noformat}
> listeners=PLAINTEXT://0.0.0.0:9092
> # advertised.listeners=PLAINTEXT://myPublicHostName:9092# REQUIRED for 
> 0.0.0.0:9092
> {noformat}
> This configuration will be invalid if {{advertised.listeners}} is not set 
> too. Connection string 0.0.0.0:9092 is stored to ZooKeeper according to 
> documentation of  {{advertised.listeners}} and observed behavior, but it 
> isn't obvious and difficult to analyze. Clients and even other brokers try to 
> communicate with brokers using destination address 0.0.0.0:9092, which is 
> impossible. Specification of {{advertised.listeners}} as shown above fixed 
> the issue.
> Please update documentation at 
> http://kafka.apache.org/0100/documentation#brokerconfigs and backport the 
> change to 0.9 and 0.10 branches.
> h4. advertised.listeners
> Listeners to publish to ZooKeeper for clients to use, if different than the 
> *`listeners`* -above-. In IaaS environments, this may need to be different 
> from the interface to which the broker binds. If this is not set, the value 
> for `listeners` will be used.
> h4. listeners
> Listener List - Comma-separated list of URIs we will listen on and their 
> protocols. Specify hostname as 0.0.0.0 to bind to all interfaces *(note 
> `advertised.listeners` configuration is required for 0.0.0.0)*. Leave 
> hostname empty to bind to default interface. Examples of legal listener 
> lists: PLAINTEXT://myhost:9092,TRACE://:9091 PLAINTEXT://0.0.0.0:9092, 
> TRACE://localhost:9093



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1974: AN-75711 Spike for Kafka server side changes to pr...

[silpamittapalli]

Github user silpamittapalli closed the pull request at:

https://github.com/apache/kafka/pull/1974


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4261) Provide debug option in vagrant-up.sh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551882#comment-15551882
 ] 

ASF GitHub Bot commented on KAFKA-4261:
---

GitHub user fpj opened a pull request:

https://github.com/apache/kafka/pull/1981

KAFKA-4261: Provide debug option in vagrant-up.sh



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fpj/kafka vagrant-debug-option

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1981


commit d39b7f4853b0c8d3d29fd3a18aeb157cec89e9c1
Author: fpj 
Date:   2016-10-01T13:58:37Z

Changes to add the debug option to vagrant-up script

commit fbdf31dfc51aae88a05a3787f268ff9d87bfe24f
Author: fpj 
Date:   2016-10-06T13:00:29Z

Fixed position of debug flag




> Provide debug option in vagrant-up.sh
> -
>
> Key: KAFKA-4261
> URL: https://issues.apache.org/jira/browse/KAFKA-4261
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Affects Versions: 0.10.1.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I found it useful to use the debug flag for vagrant, and I had to edit the 
> script to make it happen. Here I'm just proposing a simple change to provide 
> a debug command-line option instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1981: KAFKA-4261: Provide debug option in vagrant-up.sh

[fpj]

GitHub user fpj opened a pull request:

https://github.com/apache/kafka/pull/1981

KAFKA-4261: Provide debug option in vagrant-up.sh



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fpj/kafka vagrant-debug-option

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1981


commit d39b7f4853b0c8d3d29fd3a18aeb157cec89e9c1
Author: fpj 
Date:   2016-10-01T13:58:37Z

Changes to add the debug option to vagrant-up script

commit fbdf31dfc51aae88a05a3787f268ff9d87bfe24f
Author: fpj 
Date:   2016-10-06T13:00:29Z

Fixed position of debug flag




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4261) Provide debug option in vagrant-up.sh

[Flavio Junqueira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flavio Junqueira reassigned KAFKA-4261:
---

Assignee: Flavio Junqueira

> Provide debug option in vagrant-up.sh
> -
>
> Key: KAFKA-4261
> URL: https://issues.apache.org/jira/browse/KAFKA-4261
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Affects Versions: 0.10.1.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> I found it useful to use the debug flag for vagrant, and I had to edit the 
> script to make it happen. Here I'm just proposing a simple change to provide 
> a debug command-line option instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4261) Provide debug option in vagrant-up.sh

[Flavio Junqueira (JIRA)]

Flavio Junqueira created KAFKA-4261:
---

 Summary: Provide debug option in vagrant-up.sh
 Key: KAFKA-4261
 URL: https://issues.apache.org/jira/browse/KAFKA-4261
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Affects Versions: 0.10.1.0
Reporter: Flavio Junqueira
Priority: Minor
 Fix For: 0.11.0.0


I found it useful to use the debug flag for vagrant, and I had to edit the 
script to make it happen. Here I'm just proposing a simple change to provide a 
debug command-line option instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4253) Fix Kafka Stream thread shutting down process ordering

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-4253:
--
Status: Patch Available  (was: In Progress)

> Fix Kafka Stream thread shutting down process ordering
> --
>
> Key: KAFKA-4253
> URL: https://issues.apache.org/jira/browse/KAFKA-4253
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>
> Currently we close the stream thread in the following way:
> 0. Commit all tasks.
> 1. Close producer.
> 2. Close consumer.
> 3. Close restore consumer.
> 4. For each task, close its topology processors one-by-one following the 
> topology order by calling {{processor.close()}}.
> 5. For each task, close its state manager as well as flushing and closing all 
> its associated state stores.
> We choose to close the producer / consumer clients before shutting down the 
> tasks because we need to make sure all sent records has been acked so that we 
> have the right log-end-offset when closing the store and checkpointing the 
> offset of the changelog. However there is also an issue with this ordering, 
> in which users choose to write more records in their {{processor.close()}} 
> calls, this will cause RTE since the producers has already been closed, and 
> no changelog records will be able to write.
> Thinking about this issue, a more appropriate ordering will be:
> 1. For each task, close their topology processors following the topology 
> order by calling {{processor.close()}}.
> 2. For each task, commit its state by calling {{task.commit()}}. At this time 
> all sent records should be acked since {{producer.flush()}} is called.
> 3. For each task, close their {{ProcessorStateManager}}.
> 4. Close all embedded clients, i.e. producer / consumer / restore consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka clients

[Edoardo Comar]

Hi Rajini
great improvement and I see you had the code ready ... !

I now think it'd easier to implement a PR for KIP-83 (multiple logins per 
JVM) on top of yours, 
as you have now identified a client property that can be used for caching 
logins.

I'm actually wondering if I caused you to cut down your KIP !!

Also, I think your JIRA encompasses 
https://issues.apache.org/jira/browse/KAFKA-3302

thanks,
Edo
--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU



From:   Rajini Sivaram 
To: dev@kafka.apache.org
Date:   06/10/2016 10:49
Subject:[DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka 
clients



Hi all,

I have just created KIP-85 to enable JAAS login context for Kafka clients
to be configured without a physical file:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients



Comments and suggestions are welcome.

Thank you...

Regards,

Rajini



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[jira] [Commented] (KAFKA-3302) Pass kerberos keytab and principal as part of client config

[Edoardo Comar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551821#comment-15551821
 ] 

Edoardo Comar commented on KAFKA-3302:
--

looks like [~rsivaram]'s  https://issues.apache.org/jira/browse/KAFKA-4259 
includes also this issue

> Pass kerberos keytab and principal as part of client config 
> 
>
> Key: KAFKA-3302
> URL: https://issues.apache.org/jira/browse/KAFKA-3302
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ben Davison]

I also think it would be good to have a REST client that can interact with
the new management API's coming down the pipe.

On Tue, Oct 4, 2016 at 10:35 PM, Edoardo Comar  wrote:

> Harsha
> thanks for opening the discussion on this KIP.
>
> While I understand he founding members' stand that the Kafka project can
> not expand its surface to a large number of clients,
> I strongly agree with your well explained points below and support your
> KIP.
>
> A REST API is not just on the same level as any client, it's a basic
> building block of open web technologies.
> It's the API that most of our first time user want to try out (or that
> would be users ask for and expect to be there).
>
> A REST API for Kafka and a robust server implementation
> under the open governance of the Apache community would be most welcome.
>
> +1
>
> Edo
> --
> Edoardo Comar
> IBM MessageHub
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
>
> Harsha Chintalapani  wrote on 02/10/2016 21:23:15:
>
> > From: Harsha Chintalapani 
> > To: dev@kafka.apache.org
> > Date: 02/10/2016 21:23
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > Neha & Jay,
> >  We did look at the open source alternatives. Our
> concern
> > is what's the patch acceptance and adding features/ bug-fixes to the
> > existing project under a Github (although it's licensed under Apache
> 2.0).
> > It would be great if that project made available under Apache and driven
> by
> > the community.  Adding to the above, not all Kafka users are interested
> in
> > using the Java client API, they would like to have simple REST API where
> > they can code against using any language. I do believe this adds value
> to
> > Apache Kafka in itself.
> >
> > "For 1, I don't think there is value in giving in to the NIH syndrome
> and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained."
> >
> > We are not looking at this as  NIH. What we are worried about is a
> project
> > that's not maintained in a community. So the process of accepting
> patches
> > and priorities is not clear, and it's not developed in Apache community.
> > Not only that, existing REST API project doesn't support new client API
> and
> > hence there is no security support either.
> > We don't know the timeline when that's made available. We would like to
> add
> > admin functionality into the REST API. So the Roadmap of that project is
> > not driven by Apache.
> >
> >
> > "This doesn't materially have an impact on expanding the usability of
> >Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> >Kafka users, and maybe 10% of those are the ones who will use the
> REST
> >proxy. The remaining 50% are non-java client users (C, python, go,
> node
> >etc)."
> >
> > REST API is most often asked feature in my interactions with Kafka
> users.
> > In an organization, There will be independent teams who will write their
> >  Kafka clients using different language libraries available today, and
> > there is no way to standardize this. Instead of supporting several
> > different client libraries users will be interested in using a REST API
> > server. The need for a REST API will only increase as more and more
> users
> > start using Kafka.
> >
> > "More surface area means more work to keep things consistent. Failure
> >to do that has, in fact, hurt the user experience."
> > Having myriad Kafka client GitHub projects that support different
> languages
> > hurts the user experience and pushes burden to maintain these libraries.
> > REST API is a simple code base that uses existing java client libraries
> to
> > make life easier for the users.
> >
> > Thanks,
> > Harsha
> >
> > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede  wrote:
> >
> > > Manikumar,
> > >
> > > Thanks for sharing the proposal. I think there are 2 parts to this
> > > discussion -
> > >
> > > 1. Should we rewrite a REST proxy given that there is a
> feature-complete,
> > > open-source and actively maintained REST proxy in the community?
> > > 2. Does adding a REST proxy to Apache Kafka make us more agile and
> maintain
> > > the high-quality experience that Kafka users have today?
> > >
> > > For 1, I don't think there is value in giving in to the NIH syndrome
> and
> > > reinventing the wheel. What I'm looking for is a detailed comparison
> of the
> > > gaps and why those can't be improved in the REST proxy that already
> exists
> > > and is actively maintained. For example, we depend on zkClient and
> have
> > > found as well as fixed several bugs by working closely with the people
> who
> > > maintain zkClient. This should 

[jira] [Updated] (KAFKA-4252) Missing ProducerRequestPurgatory

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-4252:
--
Status: Patch Available  (was: Open)

> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.10.0.1, 0.10.0.0, 0.9.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1980: KAFKA-4252: Fix metric name in documentation

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1980

KAFKA-4252: Fix metric name in documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1980.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1980


commit 5f7dd80446263bdfd51e3c6957c98c4b422f2d9d
Author: Rajini Sivaram 
Date:   2016-10-06T11:45:16Z

KAFKA-4252: Fix metric name in documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4252) Missing ProducerRequestPurgatory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551713#comment-15551713
 ] 

ASF GitHub Bot commented on KAFKA-4252:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1980

KAFKA-4252: Fix metric name in documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1980.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1980


commit 5f7dd80446263bdfd51e3c6957c98c4b422f2d9d
Author: Rajini Sivaram 
Date:   2016-10-06T11:45:16Z

KAFKA-4252: Fix metric name in documentation




> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4252) Missing ProducerRequestPurgatory

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551696#comment-15551696
 ] 

Rajini Sivaram commented on KAFKA-4252:
---

I think the documentation was not updated when the class name changed quite a 
long time ago 
(https://github.com/apache/kafka/commit/3cc10d5ff9bf73cff2045685f89d71fee92a41f4)

> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4252) Missing ProducerRequestPurgatory

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram reassigned KAFKA-4252:
-

Assignee: Rajini Sivaram

> Missing ProducerRequestPurgatory
> 
>
> Key: KAFKA-4252
> URL: https://issues.apache.org/jira/browse/KAFKA-4252
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, documentation, metrics, producer 
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Narendra Bidari
>Assignee: Rajini Sivaram
>Priority: Minor
> Attachments: Screen Shot 2016-10-04 at 1.34.00 PM.png
>
>
> Most of the docs indicate of ProducerRequestPurgatory and 
> FetchRequestPurgatory Mbeans present in . But when I see MBean there is no 
> bean by that name.  The name which is present is *DelayedOperationPurgatory*
> are DelayedOperationPurgatory and ProducerRequestPurgatory same??
> https://github.com/apache/kafka/blob/d2a267b111d23d6b98f2784382095b9ae5ddf886/docs/ops.html
> http://docs.confluent.io/1.0/kafka/monitoring.html
> https://kafka.apache.org/08/ops.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4259) Enable JAAS configuration for Kafka clients without a config file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551615#comment-15551615
 ] 

ASF GitHub Bot commented on KAFKA-4259:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1979

KAFKA-4259: Dynamic JAAS configuration for Kafka clients

Implementation of KIP-85: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1979.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1979


commit fc9922b0d9a80053f6a88d59dc8937ea27d6c7c8
Author: Rajini Sivaram 
Date:   2016-10-06T09:51:43Z

KAFKA-4259: Dynamic JAAS configuration for Kafka clients




> Enable JAAS configuration for Kafka clients without a config file
> -
>
> Key: KAFKA-4259
> URL: https://issues.apache.org/jira/browse/KAFKA-4259
> Project: Kafka
>  Issue Type: New Feature
>  Components: security
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> See KIP-85 for details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1979: KAFKA-4259: Dynamic JAAS configuration for Kafka c...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1979

KAFKA-4259: Dynamic JAAS configuration for Kafka clients

Implementation of KIP-85: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1979.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1979


commit fc9922b0d9a80053f6a88d59dc8937ea27d6c7c8
Author: Rajini Sivaram 
Date:   2016-10-06T09:51:43Z

KAFKA-4259: Dynamic JAAS configuration for Kafka clients




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Hi All,

We have updated a sample version of an implementation of the KIP with 
Integer,Byte[] headers to aid discussion.
See:
https://github.com/michaelandrepearce/kafka/tree/headers

On the kip there are a few outstanding bits that I would like to have discussed 
and get some majority consensus on to get the KIP more complete.
I believe there is some differing on opinions on these though.

1)Should headers be ordered by the int key
2)Key Allocation – should the example be the actual proposed.

I will give my opinions on the two

1)Yes
2)Yes

===
On other notes:

@Kostya,

Thanks for the feedback. Its good seeing further people suffering the same 
issue that I haven’t already seen in stack over flow etc, giving more credence 
that this KIP has real user benefit and addresses many problems faced.

Indeed your issue on send we also have the same problem in IG with regards to 
needing to send headers with a null payload on compacting topic. We feel your 
pain here.
   • We handle this slightly differently but also with some nasty 
race/failure conditions still, currently make our producer wrapper detect this 
and creates two records one with our Message Wrapper with only headers followed 
by one with a null value. We have issues with if one record succeeds to send 
but the other doesn’t and handling this, as such it never compacts/clears out.
   • I agree this is a very bad position to be in and proves that message 
wrapper solutions doesn’t work, and one of the reasons why we want to add 
headers in this KIP ☺.

I think Radai answered your question about why Integer keys, over String based 
ones.

I agree with Radai that ordering keys whilst is an optimization, does bring 
some benefit broker side with not needing to read through all the headers.



Cheers
Mike


On 10/6/16, 2:36 AM, "Mayuresh Gharat"  wrote:

@Kostya

Regarding "To get around this we have an awful *cough* solution whereby we
have to send our message wrapper with the headers and null content, and
then we have an application that has to consume from all the compacted
topics and when it sees this message it produces back in a null payload
record to make the broker compact it out."

 ---> This has a race condition, right?

Suppose the producer produces a message with headers and null content at
time To to Kafka.

Then the producer, at time To + 1,  sends another message with headers and
actual content to Kafka.

What we expect is that the application that is consuming and then producing
same message with null payload should happen at time To + 0.5, so that the
message at To + 1 is not deleted.

But there is no guarantee here.

If the null payload goes in to Kafka at time To + 2, then essentially you
loose the second message produced by the producer at time To + 1.


Thanks,

Mayuresh

On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy  wrote:

> @Nacho
>
> > > - Brokers can't see the headers (part of the "V" black box)>
> >
>
>
> > (Also, it would be nice if we had a way to access the headers from the
> > > brokers, something that is not trivial at this time with the current
> > broker
> > > architecture).
> >
> >
>
> I think this can be addressed with broker interceptors which we touched on
> in KIP-42
>  42%3A+Add+Producer+and+Consumer+Interceptors>
> .
>
> @Gwen
>
> You are right that the wrapper thingy “works”, but there are some 
drawbacks
> that Nacho and Radai have covered in detail that I can add a few more
> comments to.
>
> At LinkedIn, we *get by* without the proposed Kafka record headers by
> dumping such metadata in one or two places:
>
>- Most of our applications use Avro, so for the most part we can use an
>explicit header field in the Avro schema. Topic owners are supposed to
>include this header in their schemas.
>- A prefix to the payload that primarily contains the schema’s ID so we
>can deserialize the Avro. (We could use this for other use-cases as
> well -
>i.e., move some of the above into this prefix blob.)
>
> Dumping headers in the Avro schema pollutes the application’s data model
> with data/service-infra-related fields that are unrelated to the 
underlying
> topic; and forces the application to deserialize the entire blob whether 
or
> not the headers are actually used. Conversely from an infrastructure
> perspective, we would really like to not touch any application data. Our
> infiltration of the application’s schema is a major reason why many at
> LinkedIn sometimes assume that we (Kafka folks) are the shepherds for all
> things Avro :)
>
> Another drawback is that all this only works if everyone in 

[DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka clients

[Rajini Sivaram]

Hi all,

I have just created KIP-85 to enable JAAS login context for Kafka clients
to be configured without a physical file:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients


Comments and suggestions are welcome.

Thank you...

Regards,

Rajini

[jira] [Updated] (KAFKA-4259) Enable JAAS configuration for Kafka clients without a config file

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-4259:
--
Description: See KIP-85 for details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients
  (was: See KIP-85 for details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration)

> Enable JAAS configuration for Kafka clients without a config file
> -
>
> Key: KAFKA-4259
> URL: https://issues.apache.org/jira/browse/KAFKA-4259
> Project: Kafka
>  Issue Type: New Feature
>  Components: security
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> See KIP-85 for details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4260) Improve documentation of configuration listeners=PLAINTEXT://0.0.0.0:9092

[Michal Turek (JIRA)]

Michal Turek created KAFKA-4260:
---

 Summary: Improve documentation of configuration 
listeners=PLAINTEXT://0.0.0.0:9092
 Key: KAFKA-4260
 URL: https://issues.apache.org/jira/browse/KAFKA-4260
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Affects Versions: 0.10.0.1
Reporter: Michal Turek
Priority: Minor


We have just updated our testing Kafka cluster to 0.10 and we were facing one 
issue with migration of legacy 0.8 configuration to

{noformat}
listeners=PLAINTEXT://0.0.0.0:9092
# advertised.listeners=PLAINTEXT://myPublicHostName:9092# REQUIRED for 
0.0.0.0:9092
{noformat}

This configuration will be invalid if {{advertised.listeners}} is not set too. 
Connection string 0.0.0.0:9092 is stored to ZooKeeper according to 
documentation of  {{advertised.listeners}} and observed behavior, but it isn't 
obvious and difficult to analyze. Clients and even other brokers try to 
communicate with brokers using destination address 0.0.0.0:9092, which is 
impossible. Specification of {{advertised.listeners}} as shown above fixed the 
issue.

Please update documentation at 
http://kafka.apache.org/0100/documentation#brokerconfigs and backport the 
change to 0.9 and 0.10 branches.

h4. advertised.listeners
Listeners to publish to ZooKeeper for clients to use, if different than the 
*`listeners`* -above-. In IaaS environments, this may need to be different from 
the interface to which the broker binds. If this is not set, the value for 
`listeners` will be used.

h4. listeners
Listener List - Comma-separated list of URIs we will listen on and their 
protocols. Specify hostname as 0.0.0.0 to bind to all interfaces *(note 
`advertised.listeners` configuration is required for 0.0.0.0)*. Leave hostname 
empty to bind to default interface. Examples of legal listener lists: 
PLAINTEXT://myhost:9092,TRACE://:9091 PLAINTEXT://0.0.0.0:9092, 
TRACE://localhost:9093



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4259) Enable JAAS configuration for Kafka clients without a config file

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-4259:
-

 Summary: Enable JAAS configuration for Kafka clients without a 
config file
 Key: KAFKA-4259
 URL: https://issues.apache.org/jira/browse/KAFKA-4259
 Project: Kafka
  Issue Type: New Feature
  Components: security
Affects Versions: 0.10.1.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 0.10.2.0


See KIP-85 for details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4258) Page not found http://kafka.apache.org/streams.html

[Andrey Dyachkov (JIRA)]

Andrey Dyachkov created KAFKA-4258:
--

 Summary: Page not found http://kafka.apache.org/streams.html
 Key: KAFKA-4258
 URL: https://issues.apache.org/jira/browse/KAFKA-4258
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Reporter: Andrey Dyachkov
Priority: Minor


It is not possible to access http://kafka.apache.org/streams.html because it 
returns:
Not Found
The requested URL /streams.html was not found on this server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1978: HOTFIX: Cannot Stop Kafka with Shell Script (Solut...

[Mabin-J]

GitHub user Mabin-J opened a pull request:

https://github.com/apache/kafka/pull/1978

HOTFIX: Cannot Stop Kafka with Shell Script (Solution 2)

If Kafka's homepath is long, kafka cannot stop with 'kafka-server-stop.sh'.

That command showed this message:
```
No kafka server to stop
```

This bug is caused that command line is too long like this.
```
/home/bbdev/Amasser/etc/alternatives/jre/bin/java -Xms1G -Xmx5G -server 
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 
-XX:+DisableExplicitGC -Djava.awt.headless=true 
-Xloggc:/home/bbdev/Amasser/var/log/kafka/kafkaServer-gc.log -verbose:gc 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Dkafka.logs.dir=/home/bbdev/Amasser/var/log/kafka 
-Dlog4j.configuration=file:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../config/log4j.properties
 -cp 
:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/argparse4j-0.5.0.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/connect-api-0.10.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/connect-file-0.10.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../li
 
bs/connect-json-0.10.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/connect-runtime-0.10.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/guava-18.0.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/hk2-api-2.4.0-b34.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-annotations-2.6.0.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-core-2.6.3.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-databind-2.6.3.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jav
 
assist-3.18.2-GA.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/javax.annotation-api-1.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/javax.inject-1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/javax.inject-2.4.0-b34.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-client-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-common-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-guava-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jersey-
 
server-2.22.2.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/jopt-simple-4.9.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/kafka_2.11-0.10.0.1.jar:/home/bbdev/Amasser/etc/alternatives/kafka/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/home/bbdev/Amasser/etc
 
/alternatives/kafka/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/home/bbdev/Amasser/etc/alternatives/kafka
```

but that is not all command line.
Full command line is this.
```
/home/bbdev/Amasser/etc/alternatives/jre/bin/java -Xms1G -Xmx5G -server 
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 
-XX:+DisableExplicitGC -Djava.awt.headless=true 
-Xloggc:/home/bbdev/Amasser/var/log/kafka/kafkaServer-gc.log -verbose:gc 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
-Dcom.sun.management.jmxremote 

[GitHub] kafka pull request #1977: HOTFIX: Cannot Stop Kafka with Shell Script (Solut...

[Mabin-J]

Github user Mabin-J closed the pull request at:

https://github.com/apache/kafka/pull/1977


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Snazzy new look to our website

[Mickael Maison]

Big improvement, it looks much better !

I noticed the Javadoc links in the API section on the documentation
page seem to be broken

On Wed, Oct 5, 2016 at 10:58 PM, Avi Flax  wrote:
>
>> On Oct 4, 2016, at 19:38, Guozhang Wang  wrote:
>>
>> The new look is great, thanks Derrick and Gwen!
>
> +1 it’s excellent!
>
>>
>> I'm wondering if we should still consider breaking "document.html" into
>> multiple pages and indexed as sub-topics on the left bar?
>
> +1 absolutely, this should _definitely_ be multiple pages. It’s pretty 
> challenging to work with this one giant page.
>
>
> 
> Software Architect @ Park Assist
> We’re hiring! http://tech.parkassist.com/jobs/
>