Kafka high cpu usage

[Andrey Dyachkov]

Hi,

Our production env uses Kafka 0.9.0.1 cluster of 12 m3.large nodes.
Partitions count per broker is ~450, percent of leaders per broker is
30-40%. The average messages load is ~3K/s, bytes flow in is ~10MB/s and
bytes flow out is ~60 MB/s.

We observed strange behaviour while putting one instance down terminating
it on AWS:

After putting down one Kafka instance, the leadership of partitions it was
a leader for was transferred to other nodes. All nodes increased their cpu
usage and one of them started consuming around 100% cpu. Restarts of that
node does not help because high cpu usage is caught up by another node.
This behaviour continues around 30 mins during that time.

In two months, we have experienced this issue several times a day.

Do you know something about that problem?
-- 

With great enthusiasm,
Andrey

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce]

Thanks guys, for discussing this offline and getting some consensus.

So its clear for myself and others what is proposed now (i think i understand, 
but want to make sure)

Could i ask either directly update the kip to detail the migration strategy, or 
(re-)state your offline discussed and agreed migration strategy based on a 
magic byte is in this thread.


The main original driver for the KIP was to support compaction where value 
isn't null, based off the discussions on KIP-82 thread.

We should be able to support non-tombstone + null value by the completion of 
the KIP, as we noted when discussing this kip, having logic based on a null 
value isn't very clean and also separates the concerns.

As discussed already though we can split this into KIP-87a and KIP-87b

Where we look to deliver KIP-87a on a compacted topic (to address the immediate 
issues)
* tombstone + null value
* tombstone + non-null value
* non-tombstone + non-null value

Then we can discuss once KIP-87a is completed options later and how we support 
the second part KIP-87b to deliver:
* non-tombstone + null value

Cheers
Mike




From: Becket Qin 
Sent: Thursday, November 17, 2016 1:43 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Renu, Mayuresh and I had an offline discussion, and following is a brief
summary.

1. We agreed that not bumping up magic value may result in losing zero copy
during migration.
2. Given that bumping up magic value is almost free and has benefit of
avoiding potential performance issue. It is probably worth doing.

One issue we still need to think about is whether we want to support a
non-tombstone message with null value.
Currently it is not supported by Kafka. If we allow a non-tombstone null
value message to exist after KIP-87. The problem is that such message will
not be supported by the consumers prior to KIP-87. Because a null value
will always be interpreted to a tombstone.

One option is that we keep the current way, i.e. do not support such
message. It would be good to know if there is a concrete use case for such
message. If there is not, we can probably just not support it.

Thanks,

JIangjie (Becket) Qin



On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat  wrote:

> Hi Ismael,
>
> This is something I can think of for migration plan:
> So the migration plan can look something like this, with up conversion :
>
> 1) Currently lets say we have Broker at version x.
> 2) Currently we have clients at version x.
> 3) a) We move the version to Broker(x+1) : supports both tombstone and null
> for log compaction.
> b) We upgrade the client to version client(x+1) : if in the producer
> client(x+1) the value is set to null, we will automatically set the
> Tombstone bit internally. If the producer client(x+1) sets the tombstone
> itself, well and good. For producer client(x), the broker will up convert
> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
> client(x+1) will be aware of this and should be able to handle this. For
> consumer client(x) we will down convert the message on the broker side.
> c) At this point we will have to specify a warning or clearly specify
> in docs that this behavior is about to be changed for log compaction.
> 4) a) In next release of the Broker(x+2), we say that only Tombstone is
> used for log compaction on the Broker side. Clients(x+1) still is
> supported.
> b) We upgrade the client to version client(x+2) : if value is set to
> null, tombstone will not be set automatically. The client will have to call
> setTombstone() to actually set the tombstone.
>
> We should compare this migration plan with the migration plan for magic
> byte bump and do whatever looks good.
> I am just worried that if we go down magic byte route, unless I am missing
> something, it sounds like kafka will be stuck with supporting both null
> value and tombstone bit for log compaction for life long, which does not
> look like a good end state.
>
> Thanks,
>
> Mayuresh
>
>
>
>
> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Ismael,
> >
> > That's a very good point which I might have not considered earlier.
> >
> > Here is a plan that I can think of:
> >
> > Stage 1) The broker from now on, up converts the message to have the
> > tombstone marker. The log compaction thread does log compaction based on
> > both null and tombstone marker. This is our transition period.
> > Stage 2) The next release we only say that log compaction is based on
> > tombstone marker. (Open source kafka makes this as a policy). By this
> time,
> > the organization which is moving to this release will be sure that they
> > have gone through the entire transition period.
> >
> > My only goal of doing this is that Kafka clearly specifies the end state
> > about what log compaction means (is it null value or a 

[GitHub] kafka pull request #2142: Merge pull request #1 from apache/trunk

[husthang]

Github user husthang closed the pull request at:

https://github.com/apache/kafka/pull/2142


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1696

[Apache Jenkins Server]

See 

Changes:

[ismael] MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION

--
[...truncated 14205 lines...]
org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #1048

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4376; Cross compile to Scala 2.12.0

--
[...truncated 3898 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #1047

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4211; Update system test services to use the new consumer by

[ismael] MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION

--
[...truncated 7947 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > 

[jira] [Assigned] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin reassigned KAFKA-4415:
---

Assignee: Dong Lin

> Reduce time to create and send MetadataUpdateRequest
> 
>
> Key: KAFKA-4415
> URL: https://issues.apache.org/jira/browse/KAFKA-4415
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> As of current implementation, when controller receives 
> ControlledShutdownRequest, it will 1) for every broker in the cluster, for 
> every partition on the broker which wants to shutdown, create an instance of 
> PartitionStateInfo and add it to 
> ControllerBrokerRequestBatch.,updateMetadataRequestMap; and 2) for every 
> broker, for every follower partitions on the broker which wants to shutdown, 
> send one MetadataUpdateRequst to that broker.
> In order to shutdown a broker, the controller will need to instantiate 
> O(partitionNum * brokerNum) PartitionStateInfo and send O(partitionNum * 
> brokerNum) partitionStateInfo. This is not efficient. The broker should only 
> need to instantiate O(partitionNum) PartitionStateInfo and send O(brokerNum) 
> MetadataUpdateRequest.
> Micro-benchmark results show that this optimization can reduce the time of 
> processing ControlledShutdownRequest by 30%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4376) Add scala 2.12 support

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4376:
---
Fix Version/s: 0.10.2.0

> Add scala 2.12 support
> --
>
> Key: KAFKA-4376
> URL: https://issues.apache.org/jira/browse/KAFKA-4376
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.10.1.0, 0.10.0.1
>Reporter: Bernard Leach
> Fix For: 0.10.2.0
>
>
> Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
> the kafka artifacts will allow downstream projects such as reactive-kafka via 
> scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4376) Add scala 2.12 support

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4376.

Resolution: Fixed

> Add scala 2.12 support
> --
>
> Key: KAFKA-4376
> URL: https://issues.apache.org/jira/browse/KAFKA-4376
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.10.1.0, 0.10.0.1
>Reporter: Bernard Leach
> Fix For: 0.10.2.0
>
>
> Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
> the kafka artifacts will allow downstream projects such as reactive-kafka via 
> scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4376) Add scala 2.12 support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672605#comment-15672605
 ] 

ASF GitHub Bot commented on KAFKA-4376:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2113


> Add scala 2.12 support
> --
>
> Key: KAFKA-4376
> URL: https://issues.apache.org/jira/browse/KAFKA-4376
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.10.1.0, 0.10.0.1
>Reporter: Bernard Leach
>
> Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
> the kafka artifacts will allow downstream projects such as reactive-kafka via 
> scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2113: KAFKA-4376: Cross compile to Scala 2.12.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2113


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2088: Cross compile to Scala 2.12.0-RC2

[leachbj]

Github user leachbj closed the pull request at:

https://github.com/apache/kafka/pull/2088


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2130: MINOR: Extract SCALA_BINARY_VERSION from SCALA_VER...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2130


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Becket Qin]

Renu, Mayuresh and I had an offline discussion, and following is a brief
summary.

1. We agreed that not bumping up magic value may result in losing zero copy
during migration.
2. Given that bumping up magic value is almost free and has benefit of
avoiding potential performance issue. It is probably worth doing.

One issue we still need to think about is whether we want to support a
non-tombstone message with null value.
Currently it is not supported by Kafka. If we allow a non-tombstone null
value message to exist after KIP-87. The problem is that such message will
not be supported by the consumers prior to KIP-87. Because a null value
will always be interpreted to a tombstone.

One option is that we keep the current way, i.e. do not support such
message. It would be good to know if there is a concrete use case for such
message. If there is not, we can probably just not support it.

Thanks,

JIangjie (Becket) Qin



On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat  wrote:

> Hi Ismael,
>
> This is something I can think of for migration plan:
> So the migration plan can look something like this, with up conversion :
>
> 1) Currently lets say we have Broker at version x.
> 2) Currently we have clients at version x.
> 3) a) We move the version to Broker(x+1) : supports both tombstone and null
> for log compaction.
> b) We upgrade the client to version client(x+1) : if in the producer
> client(x+1) the value is set to null, we will automatically set the
> Tombstone bit internally. If the producer client(x+1) sets the tombstone
> itself, well and good. For producer client(x), the broker will up convert
> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
> client(x+1) will be aware of this and should be able to handle this. For
> consumer client(x) we will down convert the message on the broker side.
> c) At this point we will have to specify a warning or clearly specify
> in docs that this behavior is about to be changed for log compaction.
> 4) a) In next release of the Broker(x+2), we say that only Tombstone is
> used for log compaction on the Broker side. Clients(x+1) still is
> supported.
> b) We upgrade the client to version client(x+2) : if value is set to
> null, tombstone will not be set automatically. The client will have to call
> setTombstone() to actually set the tombstone.
>
> We should compare this migration plan with the migration plan for magic
> byte bump and do whatever looks good.
> I am just worried that if we go down magic byte route, unless I am missing
> something, it sounds like kafka will be stuck with supporting both null
> value and tombstone bit for log compaction for life long, which does not
> look like a good end state.
>
> Thanks,
>
> Mayuresh
>
>
>
>
> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Ismael,
> >
> > That's a very good point which I might have not considered earlier.
> >
> > Here is a plan that I can think of:
> >
> > Stage 1) The broker from now on, up converts the message to have the
> > tombstone marker. The log compaction thread does log compaction based on
> > both null and tombstone marker. This is our transition period.
> > Stage 2) The next release we only say that log compaction is based on
> > tombstone marker. (Open source kafka makes this as a policy). By this
> time,
> > the organization which is moving to this release will be sure that they
> > have gone through the entire transition period.
> >
> > My only goal of doing this is that Kafka clearly specifies the end state
> > about what log compaction means (is it null value or a tombstone marker,
> > but not both).
> >
> > What do you think?
> >
> > Thanks,
> >
> > Mayuresh
> > .
> >
> > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma  wrote:
> >
> >> One comment below.
> >>
> >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> >> gharatmayures...@gmail.com
> >> > wrote:
> >>
> >> >- If we don't bump up the magic byte, on the broker side, the
> broker
> >> >will always have to look at both tombstone bit and the value when
> do
> >> the
> >> >compaction. Assuming we do not bump up the magic byte,
> >> >imagine the broker sees a message which does not have a tombstone
> bit
> >> >set. The broker does not know when the message was produced (i.e.
> >> > whether
> >> >the message has been up converted or not), it has to take a further
> >> > look at
> >> >the value to see if it is null or not in order to determine if it
> is
> >> a
> >> >tombstone. The same logic has to be put on the consumer as well
> >> because
> >> > the
> >> >consumer does not know if the message has been up converted or not.
> >> >   - If we upconvert while appending, this is not the case, right?
> >>
> >>
> >> If I understand you correctly, this is not sufficient because the log
> may
> >> have messages appended before it was upgraded to include KIP-87.
> >>
> >> 

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Ismael Juma]

Hi Mayuresh,

Thanks for describing your plan in detail. I understand the concern about
having to support both old and new way for a long time. In my opinion, we
don't have a choice in that matter. We can't change semantics of the
message format without having a long transition period. And we can't rely
on people reading documentation or acting on a warning for something so
fundamental.

As such, my take is that we need to bump the magic byte. The good news is
that we don't have to support all versions forever. We have said that we
will support direct upgrades for 2 years. That means that message format
version n could, in theory, be removed 2 years after the it's introduced.
So far, we only have 2 message format versions and we have never removed
any. So, we'll see what will actually happen, in practice.

Ismael

On Wed, Nov 16, 2016 at 9:28 PM, Mayuresh Gharat  wrote:

> Hi Ismael,
>
> This is something I can think of for migration plan:
> So the migration plan can look something like this, with up conversion :
>
> 1) Currently lets say we have Broker at version x.
> 2) Currently we have clients at version x.
> 3) a) We move the version to Broker(x+1) : supports both tombstone and null
> for log compaction.
> b) We upgrade the client to version client(x+1) : if in the producer
> client(x+1) the value is set to null, we will automatically set the
> Tombstone bit internally. If the producer client(x+1) sets the tombstone
> itself, well and good. For producer client(x), the broker will up convert
> to have the tombstone bit. Broker(x+1) is supporting both. Consumer
> client(x+1) will be aware of this and should be able to handle this. For
> consumer client(x) we will down convert the message on the broker side.
> c) At this point we will have to specify a warning or clearly specify
> in docs that this behavior is about to be changed for log compaction.
> 4) a) In next release of the Broker(x+2), we say that only Tombstone is
> used for log compaction on the Broker side. Clients(x+1) still is
> supported.
> b) We upgrade the client to version client(x+2) : if value is set to
> null, tombstone will not be set automatically. The client will have to call
> setTombstone() to actually set the tombstone.
>
> We should compare this migration plan with the migration plan for magic
> byte bump and do whatever looks good.
> I am just worried that if we go down magic byte route, unless I am missing
> something, it sounds like kafka will be stuck with supporting both null
> value and tombstone bit for log compaction for life long, which does not
> look like a good end state.
>
> Thanks,
>
> Mayuresh
>
>
>
>
> On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Ismael,
> >
> > That's a very good point which I might have not considered earlier.
> >
> > Here is a plan that I can think of:
> >
> > Stage 1) The broker from now on, up converts the message to have the
> > tombstone marker. The log compaction thread does log compaction based on
> > both null and tombstone marker. This is our transition period.
> > Stage 2) The next release we only say that log compaction is based on
> > tombstone marker. (Open source kafka makes this as a policy). By this
> time,
> > the organization which is moving to this release will be sure that they
> > have gone through the entire transition period.
> >
> > My only goal of doing this is that Kafka clearly specifies the end state
> > about what log compaction means (is it null value or a tombstone marker,
> > but not both).
> >
> > What do you think?
> >
> > Thanks,
> >
> > Mayuresh
> > .
> >
> > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma  wrote:
> >
> >> One comment below.
> >>
> >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> >> gharatmayures...@gmail.com
> >> > wrote:
> >>
> >> >- If we don't bump up the magic byte, on the broker side, the
> broker
> >> >will always have to look at both tombstone bit and the value when
> do
> >> the
> >> >compaction. Assuming we do not bump up the magic byte,
> >> >imagine the broker sees a message which does not have a tombstone
> bit
> >> >set. The broker does not know when the message was produced (i.e.
> >> > whether
> >> >the message has been up converted or not), it has to take a further
> >> > look at
> >> >the value to see if it is null or not in order to determine if it
> is
> >> a
> >> >tombstone. The same logic has to be put on the consumer as well
> >> because
> >> > the
> >> >consumer does not know if the message has been up converted or not.
> >> >   - If we upconvert while appending, this is not the case, right?
> >>
> >>
> >> If I understand you correctly, this is not sufficient because the log
> may
> >> have messages appended before it was upgraded to include KIP-87.
> >>
> >> Ismael
> >>
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
>
>
> --
> 

[jira] [Commented] (KAFKA-4211) Change system tests to use the new consumer by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672206#comment-15672206
 ] 

ASF GitHub Bot commented on KAFKA-4211:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2060


> Change system tests to use the new consumer by default
> --
>
> Key: KAFKA-4211
> URL: https://issues.apache.org/jira/browse/KAFKA-4211
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> We have utility methods like `run_produce_consume_validate` that use the old 
> consumer by default. We should change them to use the new consumer by default 
> while ensuring that we still have coverage for the old consumer (while we 
> still support it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2060: KAFKA-4211: Update system tests to use the new con...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2060


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #1045

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk7 #1694

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-4416) Add a '--group' option to the console consumer

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-4416:
--

 Summary: Add a '--group' option to the console consumer
 Key: KAFKA-4416
 URL: https://issues.apache.org/jira/browse/KAFKA-4416
 Project: Kafka
  Issue Type: Improvement
Reporter: Vahid Hashemian
Assignee: Vahid Hashemian
Priority: Minor


Add a {{--group}} option to the console consumer to simplify associating 
consumers to consumer groups. The command line option would overwrite any 
{{group.id}} property that may be specified in the consumer config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1044

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: add a space to separate two words

--
[...truncated 14368 lines...]

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetTaskWithKeyAndSerializerWhenNotRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetAllTasksWithStoreWhenNotRunning STARTED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetAllTasksWithStoreWhenNotRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartOnceClosed STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartOnceClosed PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] FAILED
java.lang.AssertionError: Condition not met within timeout 3. waiting 
for store count-by-key
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:279)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable(QueryableStateIntegrationTest.java:489)

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Mayuresh Gharat]

Hi Ismael,

This is something I can think of for migration plan:
So the migration plan can look something like this, with up conversion :

1) Currently lets say we have Broker at version x.
2) Currently we have clients at version x.
3) a) We move the version to Broker(x+1) : supports both tombstone and null
for log compaction.
b) We upgrade the client to version client(x+1) : if in the producer
client(x+1) the value is set to null, we will automatically set the
Tombstone bit internally. If the producer client(x+1) sets the tombstone
itself, well and good. For producer client(x), the broker will up convert
to have the tombstone bit. Broker(x+1) is supporting both. Consumer
client(x+1) will be aware of this and should be able to handle this. For
consumer client(x) we will down convert the message on the broker side.
c) At this point we will have to specify a warning or clearly specify
in docs that this behavior is about to be changed for log compaction.
4) a) In next release of the Broker(x+2), we say that only Tombstone is
used for log compaction on the Broker side. Clients(x+1) still is
supported.
b) We upgrade the client to version client(x+2) : if value is set to
null, tombstone will not be set automatically. The client will have to call
setTombstone() to actually set the tombstone.

We should compare this migration plan with the migration plan for magic
byte bump and do whatever looks good.
I am just worried that if we go down magic byte route, unless I am missing
something, it sounds like kafka will be stuck with supporting both null
value and tombstone bit for log compaction for life long, which does not
look like a good end state.

Thanks,

Mayuresh




On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat  wrote:

> Hi Ismael,
>
> That's a very good point which I might have not considered earlier.
>
> Here is a plan that I can think of:
>
> Stage 1) The broker from now on, up converts the message to have the
> tombstone marker. The log compaction thread does log compaction based on
> both null and tombstone marker. This is our transition period.
> Stage 2) The next release we only say that log compaction is based on
> tombstone marker. (Open source kafka makes this as a policy). By this time,
> the organization which is moving to this release will be sure that they
> have gone through the entire transition period.
>
> My only goal of doing this is that Kafka clearly specifies the end state
> about what log compaction means (is it null value or a tombstone marker,
> but not both).
>
> What do you think?
>
> Thanks,
>
> Mayuresh
> .
>
> On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma  wrote:
>
>> One comment below.
>>
>> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
>> gharatmayures...@gmail.com
>> > wrote:
>>
>> >- If we don't bump up the magic byte, on the broker side, the broker
>> >will always have to look at both tombstone bit and the value when do
>> the
>> >compaction. Assuming we do not bump up the magic byte,
>> >imagine the broker sees a message which does not have a tombstone bit
>> >set. The broker does not know when the message was produced (i.e.
>> > whether
>> >the message has been up converted or not), it has to take a further
>> > look at
>> >the value to see if it is null or not in order to determine if it is
>> a
>> >tombstone. The same logic has to be put on the consumer as well
>> because
>> > the
>> >consumer does not know if the message has been up converted or not.
>> >   - If we upconvert while appending, this is not the case, right?
>>
>>
>> If I understand you correctly, this is not sufficient because the log may
>> have messages appended before it was upgraded to include KIP-87.
>>
>> Ismael
>>
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[GitHub] kafka pull request #2124: KAFKA-4359: Removed commit interval

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2124


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4359) Streams integration tests should not use commit interval of 1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671584#comment-15671584
 ] 

ASF GitHub Bot commented on KAFKA-4359:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2124


> Streams integration tests should not use commit interval of 1
> -
>
> Key: KAFKA-4359
> URL: https://issues.apache.org/jira/browse/KAFKA-4359
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.1
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.2.0
>
>
> Several streams integration tests use two cache sizes, 0 and 10MB. However, 
> when they use 10MB, they still use a very small commit interval (1ms). That 
> leads to two problems:1) a small commit interval often has the same effect as 
> having the cache size be 0, and 2) a small commit interval is not exactly the 
> same as the cache size being 0 and in some cases there is deduplication. This 
> leads to the tests failing, since they don't expect deduplication.
> To solve this issue, look at KStreamAggregationDedupIntegrationTest and 
> KStreamAggregationIntegrationTest. If you want to test dedup, it would be 
> necessary to create another file. 
> Several tests need this cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4359) Streams integration tests should not use commit interval of 1

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4359.
--
   Resolution: Fixed
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

Issue resolved by pull request 2124
[https://github.com/apache/kafka/pull/2124]

> Streams integration tests should not use commit interval of 1
> -
>
> Key: KAFKA-4359
> URL: https://issues.apache.org/jira/browse/KAFKA-4359
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.1
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.2.0
>
>
> Several streams integration tests use two cache sizes, 0 and 10MB. However, 
> when they use 10MB, they still use a very small commit interval (1ms). That 
> leads to two problems:1) a small commit interval often has the same effect as 
> having the cache size be 0, and 2) a small commit interval is not exactly the 
> same as the cache size being 0 and in some cases there is deduplication. This 
> leads to the tests failing, since they don't expect deduplication.
> To solve this issue, look at KStreamAggregationDedupIntegrationTest and 
> KStreamAggregationIntegrationTest. If you want to test dedup, it would be 
> necessary to create another file. 
> Several tests need this cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[Mark Shelton (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671564#comment-15671564
 ] 

Mark Shelton commented on KAFKA-4322:
-

I have a custom store and processor and if no information are available to the 
store and/or processor about whether or what was restored that impacts its 
usefulness. We are using our own logging. Relying on just the Kafka metrics or 
logging would be awkward or even insufficient.


> StateRestoreCallback begin and end indication
> -
>
> Key: KAFKA-4322
> URL: https://issues.apache.org/jira/browse/KAFKA-4322
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Mark Shelton
>Assignee: Mark Shelton
>Priority: Minor
>
> In Kafka Streams, the StateRestoreCallback interface provides only a single 
> method "restore(byte[] key, byte[] value)" that is called for every key-value 
> pair to be restored. 
> It would be nice to have "beginRestore" and "endRestore" methods as part of 
> StateRestoreCallback.
> Kafka Streams would call "beginRestore" before restoring any keys, and would 
> call "endRestore" when it determines that it is done. This allows an 
> implementation, for example, to report on the number of keys restored and 
> perform a commit after the last key was restored. Other uses are conceivable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1693

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: add a space to separate two words

--
[...truncated 3898 lines...]
kafka.server.DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties STARTED

kafka.server.DynamicConfigChangeTest > 
shouldParseWildcardReplicationQuotaProperties PASSED

kafka.server.DynamicConfigChangeTest > testDefaultClientIdQuotaConfigChange 
STARTED

kafka.server.DynamicConfigChangeTest > testDefaultClientIdQuotaConfigChange 
PASSED

kafka.server.DynamicConfigChangeTest > testQuotaInitialization STARTED

kafka.server.DynamicConfigChangeTest > testQuotaInitialization PASSED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange STARTED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange PASSED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
STARTED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
PASSED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues STARTED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues PASSED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaReset STARTED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaReset PASSED

kafka.server.DynamicConfigChangeTest > testDefaultUserClientIdQuotaConfigChange 
STARTED

kafka.server.DynamicConfigChangeTest > testDefaultUserClientIdQuotaConfigChange 
PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic 
STARTED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod STARTED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod PASSED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision PASSED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps PASSED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK STARTED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest FAILED
org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: 
Could not find a 'KafkaServer' entry in the JAAS configuration. System property 
'java.security.auth.login.config' is /tmp/kafka4261226622793356690.tmp
at 
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at 
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
at kafka.network.Processor.(SocketServer.scala:406)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:141)
at 

[jira] [Commented] (KAFKA-4400) Prefix for sink task consumer groups should be configurable

[Tianji Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671457#comment-15671457
 ] 

Tianji Li commented on KAFKA-4400:
--

[~ewencp] We got bugged by this very issue a few times and so I did a PR to fix 
it. Please let me know if it is OK.

> Prefix for sink task consumer groups should be configurable
> ---
>
> Key: KAFKA-4400
> URL: https://issues.apache.org/jira/browse/KAFKA-4400
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>
> Currently the prefix for creating consumer groups is fixed. This means that 
> if you run multiple Connect clusters using the same Kafka cluster and create 
> connectors with the same name, sink tasks in different clusters will join the 
> same group. Making this prefix configurable at the worker level would protect 
> against this.
> An alternative would be to define unique cluster IDs for each connect 
> cluster, which would allow us to construct a unique name for the group 
> without requiring yet another config (but presents something of a 
> compatibility challenge).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4400) Prefix for sink task consumer groups should be configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671433#comment-15671433
 ] 

ASF GitHub Bot commented on KAFKA-4400:
---

GitHub user skyahead opened a pull request:

https://github.com/apache/kafka/pull/2143

KAFKA-4400: Enabling configurable prefix for sink connectors' consume…

…r group names.

Author: Tianji Li 

Reviewers: Ewen Cheslack-Postava 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/skyahead/kafka KAFKA4400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2143






> Prefix for sink task consumer groups should be configurable
> ---
>
> Key: KAFKA-4400
> URL: https://issues.apache.org/jira/browse/KAFKA-4400
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>
> Currently the prefix for creating consumer groups is fixed. This means that 
> if you run multiple Connect clusters using the same Kafka cluster and create 
> connectors with the same name, sink tasks in different clusters will join the 
> same group. Making this prefix configurable at the worker level would protect 
> against this.
> An alternative would be to define unique cluster IDs for each connect 
> cluster, which would allow us to construct a unique name for the group 
> without requiring yet another config (but presents something of a 
> compatibility challenge).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2143: KAFKA-4400: Enabling configurable prefix for sink ...

[skyahead]

GitHub user skyahead opened a pull request:

https://github.com/apache/kafka/pull/2143

KAFKA-4400: Enabling configurable prefix for sink connectors' consume…

…r group names.

Author: Tianji Li 

Reviewers: Ewen Cheslack-Postava 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/skyahead/kafka KAFKA4400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2143






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2132: add a space to separate two words

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2132


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671351#comment-15671351
 ] 

ASF GitHub Bot commented on KAFKA-4089:
---

Github user sutambe closed the pull request at:

https://github.com/apache/kafka/pull/1791


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Sumant Tambe
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally restrict 
> ejection when metadata is stale. 
> Conversely, it should expire batches only when the following is true
> # !muted AND
> # meta-data is fresh AND
> # batch remained in the queue longer than request timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1791: KAFKA-4089: KafkaProducer expires batch when metad...

[sutambe]

Github user sutambe closed the pull request at:

https://github.com/apache/kafka/pull/1791


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #1043

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Remove unused `ByteBoundedBlockingQueue` class and

--
[...truncated 6405 lines...]

kafka.api.AdminClientTest > testDescribeConsumerGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroup PASSED

kafka.api.AdminClientTest > testListGroups STARTED

kafka.api.AdminClientTest > testListGroups PASSED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup STARTED

kafka.api.AdminClientTest > testDescribeConsumerGroupForNonExistentGroup PASSED

kafka.api.AdminClientTest > testGetConsumerGroupSummary STARTED

kafka.api.AdminClientTest > testGetConsumerGroupSummary PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoConsumeWithDescribeAclViaAssign 
PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl PASSED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe STARTED

kafka.api.SslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe PASSED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl STARTED

kafka.api.SslEndToEndAuthorizationTest > testNoProduceWithoutDescribeAcl PASSED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic STARTED

kafka.api.RackAwareAutoTopicCreationTest > testAutoCreateTopic PASSED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets STARTED

kafka.api.PlaintextConsumerTest > testEarliestOrLatestOffsets PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs PASSED

kafka.api.PlaintextConsumerTest > testOffsetsForTimes STARTED

kafka.api.PlaintextConsumerTest > testOffsetsForTimes PASSED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription STARTED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testAsyncCommit STARTED

kafka.api.PlaintextConsumerTest > testAsyncCommit PASSED

kafka.api.PlaintextConsumerTest > testLowMaxFetchSizeForRequestAndPartition 
STARTED

kafka.api.PlaintextConsumerTest > testLowMaxFetchSizeForRequestAndPartition 
PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
PASSED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMsDelayInRevocation STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMsDelayInRevocation PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance STARTED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst STARTED

kafka.api.PlaintextConsumerTest > 
testFetchHonoursFetchSizeIfLargeRecordNotFirst PASSED

kafka.api.PlaintextConsumerTest > testSeek STARTED

kafka.api.PlaintextConsumerTest > testSeek PASSED

kafka.api.PlaintextConsumerTest > testPositionAndCommit STARTED

kafka.api.PlaintextConsumerTest > testPositionAndCommit PASSED

kafka.api.PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes STARTED

kafka.api.PlaintextConsumerTest > 
testFetchRecordLargerThanMaxPartitionFetchBytes PASSED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic STARTED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose STARTED

kafka.api.PlaintextConsumerTest > 

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Guozhang Wang]

David,

I think it would be better implementing such synchronization (i.e. making
sure all consumers has done fetching to that point, and no one will ever
want to go back and re-consume) on the admin side, not on the broker side,
since 1) we want to keep the broker system to be simple enough, and rather
have a "layered architecture" to have such admin features on-top / by-side
of the brokers rather built inside it, and 2) for some synchronization
purposes like "making sure no on will ever want to go back and re-consume",
brokers would not have any clues and it needs to be implemented from
application to application anyways.

What do you think?

Guozhang



On Sun, Nov 13, 2016 at 6:16 AM, 东方甲乙 <254479...@qq.com> wrote:

> Hi Becket,
> If using the trim.on.offset.commit parameter,  it will help to quickly
> trim the log, but other consumer group's consumer may find the messages are
> trimmed.
> We still need to coordinate many consumer groups to trim the log, it seems
> difficult for the single consumer to do it.
> Then it will still come to the problem: whether to implement in the
> broker side or in the admin client side.  Even implement in the broker
> side, we can still using the
> trim API to finish the log deletion for Leader or Replica segments.  And
> we can offer an option to safely delete the log(disable by default), so
> this is motivation for this KIP.
>
>
> Thanks,
> David
>
>
>
>
>
>
>
> -- 原始邮件 --
> 发件人: "Becket Qin";;
> 发送时间: 2016年11月6日(星期天) 晚上11:39
> 收件人: "dev";
>
> 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
>
>
>
> Hi David,
>
> I am thinking that depending on the use case, we may not need a separate
> tool to have the committed message based retention using the trim() method.
> One way to do this is to have a configuration like trim.on.offset.commit in
> the consumer so after committing the offset, the consumer will also send a
> trim request to the broker.
>
> In some cases, the application may want to trim the log in a more flexible
> way, e.g not trim on commit but every hour. In that case, it is true that
> users will need to trim the log with a separate admin client. However that
> logic could be a long running stand-alone service independent of Kafka or
> the application. It may have its own configurations as we discussed in this
> KIP so the applications in that case would just talk to that service to
> trim the log instead of taking to Kafka.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Sun, Nov 6, 2016 at 6:10 AM, 东方甲乙 <254479...@qq.com> wrote:
>
> > Hi Becket,
> > The most important benefit of method (2) is we can safely delete the
> > log segments, becasue all the deleted log segments are consumed.
> > If the messages  are very important, in this case we need to safely
> delete
> > the log segments instead of forcing delete it after the retention time.
> > Kafka itself can insure all the deleted logs are consumed to improve
> > End-to-End reliability.  And this feature by default is disabled, so will
> > stay simple for people not use it.
> > Actually users can build a tool using the trimRequest to do this
> > work(method 1), but users must start this tool with kafka all the time,
> > this may not always holds.
> >
> >
> > Thanks,
> > David
> >
> >
> >
> >
> >
> >
> >
> >
> > -- 原始邮件 --
> > 发件人: "Becket Qin";;
> > 发送时间: 2016年11月1日(星期二) 凌晨3:57
> > 收件人: "dev";
> >
> > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log
> retention
> >
> >
> >
> > Hi David,
> >
> > I think the trim() API is generally useful for the consume based
> retention
> > as well as other use cases. So we probably should have (1).
> >
> > For (2), it is more of an optimization by doing a favor for the users.
> This
> > could be implemented on top of (1) if we want to. So maybe we can
> implement
> > (1) first and let the applications do the trim() by themselves at this
> > point. This will put more burden on the application side but is not that
> > bad if there is only one downstream consumer group. In the future if we
> > find more use cases where multiple down stream consumer groups need to
> > coordinate among themselves and a broker side help would make things
> > simpler, we can add (2) then.
> >
> > Regarding the relation between this KIP and KIP-47. At a high level, they
> > are very similar, i.e. trim() by timestamp vs. trim() by offsets. It
> would
> > be worth thinking about them together. After KIP-79, we can search
> messages
> > by timestamp, this essentially translates the timestamp to offsets. So
> > KIP-47 can also be built on top of the trim() by offsets interface after
> > translating the timestamp to offsets. Jun has suggested an implementation
> > in KIP-47 discussion thread which introduces a new TrimRequest. Would you
> > take a look and see if that could be used for KIP-68 

[jira] [Commented] (KAFKA-4414) Unexpected "Halting because log truncation is not allowed"

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671200#comment-15671200
 ] 

James Cheng commented on KAFKA-4414:


Possibly related to https://issues.apache.org/jira/browse/KAFKA-3410. There is 
more info over there.

> Unexpected "Halting because log truncation is not allowed"
> --
>
> Key: KAFKA-4414
> URL: https://issues.apache.org/jira/browse/KAFKA-4414
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Meyer Kizner
>
> Our Kafka installation runs with unclean leader election disabled, so brokers 
> halt when they find that their message offset is ahead of the leader's offset 
> for a topic. We had two brokers halt today with this issue. After much time 
> spent digging through the logs, I believe the following timeline describes 
> what occurred and points to a plausible hypothesis as to what happened.
> * B1, B2, and B3 are replicas of a topic, all in the ISR. B2 is currently the 
> leader, but B1 is the preferred leader. The controller runs on B3.
> * B1 fails, but the controller does not detect the failure immediately.
> * B2 receives a message from a producer and B3 fetches it to stay up to date. 
> B2 has not accepted the message, because B1 is down and so has not 
> acknowledged the message.
> * The controller triggers a preferred leader election, making B1 the leader, 
> and notifies all replicas.
> * Very shortly afterwards (~200ms), B1's broker registration in ZooKeeper 
> expires, so the controller reassigns B2 to be leader again and notifies all 
> replicas.
> * Because B3 is the controller, while B2 is on another box, B3 hears about 
> both of these events before B2 hears about either. B3 truncates its log to 
> the high water mark (before the pending message) and resumes fetching from B2.
> * B3 fetches the pending message from B2 again.
> * B2 learns that it has been displaced and then reelected, and truncates its 
> log to the high water mark, before the pending message.
> * The next time B3 tries to fetch from B2, it sees that B2 is missing the 
> pending message and halts.
> In this case, there was no data loss or inconsistency. I haven't fully 
> thought through whether either would be possible, but it seems likely that 
> they would be, especially if there had been multiple producers to this topic.
> I'm not completely certain about this timeline, but this sequence of events 
> appears to at least be possible. Looking a bit through the controller code, 
> there doesn't seem to be anything that forces {{LeaderAndIsrRequest}} to be 
> sent in a particular order. If someone with more knowledge of the code base 
> believes this is incorrect, I'd be happy to post the logs and/or do some more 
> digging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2136: MINOR: Remove unused `ByteBoundedBlockingQueue` cl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2136


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671093#comment-15671093
 ] 

Guozhang Wang commented on KAFKA-4322:
--

Thanks [~markshelton], the reason I mentioned a KIP proposal is that we seem 
have different opinions (having a callback, or rather just adding the metrics / 
log entries) on this issue, and KIP discussion was designed for such cases 
where people can speak out openly and bring community's attention to any 
changes to public APIs.

Given your use case, I'm wondering if it would OK to just adding logging to 
record how many keys were restored and also metrics regarding restoration, 
instead of adding a generalized callback? If in the future we observed other 
use cases that can be captured in a callback I'm more than happy to propose the 
KIP myself and push this change to public APIs, since after all it is easier to 
add a new function later than adding the function now and reverting it for 
simplicity.

> StateRestoreCallback begin and end indication
> -
>
> Key: KAFKA-4322
> URL: https://issues.apache.org/jira/browse/KAFKA-4322
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Mark Shelton
>Assignee: Mark Shelton
>Priority: Minor
>
> In Kafka Streams, the StateRestoreCallback interface provides only a single 
> method "restore(byte[] key, byte[] value)" that is called for every key-value 
> pair to be restored. 
> It would be nice to have "beginRestore" and "endRestore" methods as part of 
> StateRestoreCallback.
> Kafka Streams would call "beginRestore" before restoring any keys, and would 
> call "endRestore" when it determines that it is done. This allows an 
> implementation, for example, to report on the number of keys restored and 
> perform a commit after the last key was restored. Other uses are conceivable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Mayuresh Gharat]

Hi Ismael,

That's a very good point which I might have not considered earlier.

Here is a plan that I can think of:

Stage 1) The broker from now on, up converts the message to have the
tombstone marker. The log compaction thread does log compaction based on
both null and tombstone marker. This is our transition period.
Stage 2) The next release we only say that log compaction is based on
tombstone marker. (Open source kafka makes this as a policy). By this time,
the organization which is moving to this release will be sure that they
have gone through the entire transition period.

My only goal of doing this is that Kafka clearly specifies the end state
about what log compaction means (is it null value or a tombstone marker,
but not both).

What do you think?

Thanks,

Mayuresh
.

On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma  wrote:

> One comment below.
>
> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> >- If we don't bump up the magic byte, on the broker side, the broker
> >will always have to look at both tombstone bit and the value when do
> the
> >compaction. Assuming we do not bump up the magic byte,
> >imagine the broker sees a message which does not have a tombstone bit
> >set. The broker does not know when the message was produced (i.e.
> > whether
> >the message has been up converted or not), it has to take a further
> > look at
> >the value to see if it is null or not in order to determine if it is a
> >tombstone. The same logic has to be put on the consumer as well
> because
> > the
> >consumer does not know if the message has been up converted or not.
> >   - If we upconvert while appending, this is not the case, right?
>
>
> If I understand you correctly, this is not sufficient because the log may
> have messages appended before it was upgraded to include KIP-87.
>
> Ismael
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[jira] [Commented] (KAFKA-4391) On Windows, Kafka server stops with uncaught exception after coming back from sleep

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671078#comment-15671078
 ] 

Guozhang Wang commented on KAFKA-4391:
--

Currently we do not have plans to release new bug fix versions to 0.9.x yet, 
since the community tend to encourage people to upgrade to newer major / minor 
versions than backporting bug fixes. But if you would like to back port this 
change back to 0.9.x I can help merge it, not sure if / when a new bug fix 
release of 0.9.x will be out though.

> On Windows, Kafka server stops with uncaught exception after coming back from 
> sleep
> ---
>
> Key: KAFKA-4391
> URL: https://issues.apache.org/jira/browse/KAFKA-4391
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
> Environment: Windows 10, jdk1.8.0_111
>Reporter: Yiquan Zhou
>
> Steps to reproduce:
> 1. start the zookeeper
> $ bin\windows\zookeeper-server-start.bat config/zookeeper.properties
> 2. start the Kafka server with the default properties
> $ bin\windows\kafka-server-start.bat config/server.properties
> 3. put Windows into sleep mode for 1-2 hours
> 4. activate Windows again, an exception occurs in Kafka server console and 
> the server is stopped:
> {code:title=kafka console log}
> [2016-11-08 21:45:35,185] INFO Client session timed out, have not heard from 
> server in 10081379ms for sessionid 0x1584514da47, closing socket 
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:40,698] INFO zookeeper state changed (Disconnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-11-08 21:45:43,029] INFO Opening socket connection to server 
> 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL 
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,044] INFO Socket connection established to 
> 127.0.0.1/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,158] INFO Unable to reconnect to ZooKeeper service, 
> session 0x1584514da47 has expired, closing socket connection 
> (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,158] INFO zookeeper state changed (Expired) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-11-08 21:45:43,236] INFO Initiating client connection, 
> connectString=localhost:2181 sessionTimeout=6000 
> watcher=org.I0Itec.zkclient.ZkClient@11ca437b (org.apache.zookeeper.ZooKeeper)
> [2016-11-08 21:45:43,280] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> log4j:ERROR Failed to rename [/controller.log] to 
> [/controller.log.2016-11-08-18].
> [2016-11-08 21:45:43,421] INFO Opening socket connection to server 
> 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL 
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,483] INFO Socket connection established to 
> 127.0.0.1/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,811] INFO Session establishment complete on server 
> 127.0.0.1/127.0.0.1:2181, sessionid = 0x1584514da470001, negotiated timeout = 
> 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,827] INFO zookeeper state changed (SyncConnected) 
> (org.I0Itec.zkclient.ZkClient)
> log4j:ERROR Failed to rename [/server.log] to [/server.log.2016-11-08-18].
> [2016-11-08 21:45:43,827] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-11-08 21:45:44,014] INFO Result of znode creation is: OK 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-11-08 21:45:44,014] INFO 0 successfully elected as leader 
> (kafka.server.ZookeeperLeaderElector)
> log4j:ERROR Failed to rename [/state-change.log] to 
> [/state-change.log.2016-11-08-18].
> [2016-11-08 21:45:44,421] INFO re-registering broker info in ZK for broker 0 
> (kafka.server.KafkaHealthcheck)
> [2016-11-08 21:45:44,436] INFO Creating /brokers/ids/0 (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-11-08 21:45:44,686] INFO Result of znode creation is: OK 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-11-08 21:45:44,686] INFO Registered broker 0 at path /brokers/ids/0 
> with addresses: PLAINTEXT -> EndPoint(192.168.0.15,9092,PLAINTEXT) 
> (kafka.utils.ZkUtils)
> [2016-11-08 21:45:44,686] INFO done re-registering broker 
> (kafka.server.KafkaHealthcheck)
> [2016-11-08 21:45:44,686] INFO Subscribing to /brokers/topics path to watch 
> for new topics (kafka.server.KafkaHealthcheck)
> [2016-11-08 21:45:45,046] INFO [ReplicaFetcherManager on broker 0] Removed 
> fetcher for partitions [test,0] (kafka.server.ReplicaFetcherManager)
> [2016-11-08 21:45:45,061] INFO New leader is 0 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2016-11-08 21:45:47,325] ERROR Uncaught exception in scheduled task 
> 

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Ismael Juma]

One comment below.

On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat  wrote:

>- If we don't bump up the magic byte, on the broker side, the broker
>will always have to look at both tombstone bit and the value when do the
>compaction. Assuming we do not bump up the magic byte,
>imagine the broker sees a message which does not have a tombstone bit
>set. The broker does not know when the message was produced (i.e.
> whether
>the message has been up converted or not), it has to take a further
> look at
>the value to see if it is null or not in order to determine if it is a
>tombstone. The same logic has to be put on the consumer as well because
> the
>consumer does not know if the message has been up converted or not.
>   - If we upconvert while appending, this is not the case, right?


If I understand you correctly, this is not sufficient because the log may
have messages appended before it was upgraded to include KIP-87.

Ismael

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Mayuresh Gharat]

Sorry for sending this late.

Please see the replies/comments inline :

   - In practice, the up-conversion may not be used until all the clients
   are upgraded.
  - I am not sure what you meant by this. Up conversion is actually
  done for old producer clients producing to the new broker only, right? So
  we don't have to wait for all clients to be upgraded.
   - When rolling out the new server code, we will keep the message format
   at 0.10.1. We cannot use message format version 0.10.2 yet because if we do
   that the brokers will suddenly lose zero copy.
  - I am not sure if this is entirely true. There is no notion of zero
  copy on the produce side, I think (and as per Joel). I am not
100% familiar
  with this part but from the first look at the code, it does not seem to
  have a notion of zero copy.
   - If we bump up the message version to 0.10.2, the broker will have to
   look at the message to see if down conversion is needed).
   - I think Renu was talking about the request version and not the message
  version. The down conversion is anyways needed for the new broker talking
  to old consumer anyways.
   - Later on when most of the consumers have upgraded, we will then bump
   up the message format version to 0.10.2.
  - I am not sure what you mean by most of the consumers. The way I
  look at it is, either all consumers have upgraded then we don't have to
  down convert or else we will have to down convert.
   - If we don't bump up the magic byte, on the broker side, the broker
   will always have to look at both tombstone bit and the value when do the
   compaction. Assuming we do not bump up the magic byte,
   imagine the broker sees a message which does not have a tombstone bit
   set. The broker does not know when the message was produced (i.e. whether
   the message has been up converted or not), it has to take a further look at
   the value to see if it is null or not in order to determine if it is a
   tombstone. The same logic has to be put on the consumer as well because the
   consumer does not know if the message has been up converted or not.
  - If we upconvert while appending, this is not the case, right? The
  consumer will only look at the tombstone marker. Actually the consumers
  don't care about the value is null or tombstone marker is set, they just
  consume whatever is there on the broker right? It is the
applications that
  are using the consumer that will care. And since in the second stage, the
  applications will be aware of the behavior, I think it should be
fine. Also
  the broker does not need to know when the message was produced
since at the
  time of appending it knows what the request version was and since it has
  already upconverted the message, if it gets a fetch request from old
  consumer it will down convert (as you have suggested earlier) but for the
  fetch new request version it will not do anything. During log compaction,
  it will only take a look at the tombstone marker.
  - As per the documentation of when to up the magic byte, it should be
  done only during message format change. This is not ideally a message
  format change right, since we are just using the reserved bit? If that is
  not the case, then we should change the description of when to
up the magic
  byte to something like any changes made to existing message format
  (addition of new fields or using existing reserved fields)

Thanks,

Mayuresh


On Tue, Nov 15, 2016 at 3:35 AM, Becket Qin  wrote:

> Hi Renu,
>
> Technically speaking we may not need to bump up the magic value. A
> tombstone is a message either with tombstone bit set OR with a null
> value.(once all the clients has been upgraded, it automatically becomes
> only based on tombstone) However, this leads to a few subtle issues.
>
> 1. Migration. In practice, the up-conversion may not be used until all the
> clients are upgraded. Kafka requires the client version to be lower than
> the server version for compatibility. That means the new server code is
> always deployed when clients are still running old code. When rolling out
> the new server code, we will keep the message format at 0.10.1. We cannot
> use message format version 0.10.2 yet because if we do that the brokers
> will suddenly lose zero copy This is because all the clients are still
> running old code. If we bump up the message version to 0.10.2, the broker
> will have to look at the message to see if down conversion is needed).
> Later on when most of the consumers have upgraded, we will then bump up the
> message format version to 0.10.2. So the broker cannot always up convert
> and depend on the tombstone even with the new code.
>
> 2. Long term efficiency. If we don't bump up the magic byte, on the broker
> side, the broker will always have to look at both tombstone bit and the
> value when do the compaction. 

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

[Jun Rao]

Hi, Radai,

Thanks for the updated proposal. +1 overall. A couple of comments below.

1. Our current convention is to avoid using getters. Could you change
getSize and getAvailableMemory accordingly? Also, size is bit ambiguous,
could we use sth like capacity?

2. This is more on the implementation details. I didn't see any code to
wake up the selector when memory is released from the pool. For example,
suppose that all socket keys are muted since the pool is full. The
selector.poll() call will wait for the timeout, which could be arbitrarily
long. Now, if some memory is released, it seems that we should wake up the
selector early instead of waiting for the timeout.

Jun


On Mon, Nov 14, 2016 at 11:41 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> +1
>
> Thank you for the KIP, Radai.
>
> On Mon, Nov 14, 2016 at 6:07 PM, Mickael Maison 
> wrote:
>
> > +1. We've also been hit by OOMs on the broker because we were not able
> > to properly bound its memory usage.
> >
> > On Mon, Nov 14, 2016 at 5:56 PM, radai 
> wrote:
> > > @rajini - fixed the hasBytesBuffered() method. also updated poll() so
> > that
> > > no latency is added for picking up data stuck in ssl buffers (timeout
> is
> > > set to 0, just like with immediately connected keys and staged
> receives).
> > > thank you for pointing these out.
> > > added ssl (re) testing to the KIP testing plan.
> > >
> > >
> > >
> > >
> > > On Mon, Nov 14, 2016 at 7:24 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > >> Open point 1. I would just retain the current long value that
> specifies
> > >> queued.max.bytes as long and not as %heap since it is simple and easy
> to
> > >> use. And keeps it consistent with other ".bytes" configs.
> > >>
> > >> Point 3. ssl buffers - I am not quite sure the implementation looks
> > >> correct. hasBytesBuffered() is checking position() of buffers == 0.
> And
> > the
> > >> code checks this only when poll with a timeout returns (adding a delay
> > when
> > >> there is nothing else to read).
> > >> But since this and open point 2 (optimization) are implementation
> > details,
> > >> they can be looked at during PR review.
> > >>
> > >> It will be good to add SSL testing to the test plan as well, since
> > there is
> > >> additional code to test for SSL.
> > >>
> > >>
> > >> On Fri, Nov 11, 2016 at 9:03 PM, radai 
> > wrote:
> > >>
> > >> > ok, i've made the following changes:
> > >> >
> > >> > 1. memory.pool.class.name has been removed
> > >> > 2. the code now only uses SimpleMemoryPool. the gc variant is left
> > >> (unused)
> > >> > as a developement aid and is unsettable via configuration.
> > >> > 3. I've resolved the issue of stale data getting stuck in
> intermediate
> > >> > (ssl) buffers.
> > >> > 4. default value for queued.max.bytes is -1, so off by default. any
> > <=0
> > >> > value is interpreted as off by the underlying code.
> > >> >
> > >> > open points:
> > >> >
> > >> > 1. the kafka config framework doesnt allow a value to be either long
> > or
> > >> > double, so in order to pull off the queued.max.bytes = 100 or
> > >> > queued.max.bytes = 0.3 thing i'd need to define the config as type
> > >> string,
> > >> > which is ugly to me. do we want to support setting queued.max.bytes
> > to %
> > >> of
> > >> > heap ? if so, by way of making queued.max.bytes of type string, or
> by
> > way
> > >> > of a 2nd config param (with the resulting either/all/combination?
> > >> > validation). my personal opinion is string because i think a single
> > >> > queued.max.bytes with overloaded meaning is more understandable to
> > users.
> > >> > i'll await other people's opinions before doing anything.
> > >> > 2. i still need to evaluate rajini's optimization. sounds doable.
> > >> >
> > >> > asides:
> > >> >
> > >> > 1. i think you guys misunderstood the intent behind the gc pool. it
> > was
> > >> > never meant to be a magic pool that automatically releases buffers
> > >> (because
> > >> > just as rajini stated the performance implications would be
> > horrible). it
> > >> > was meant to catch leaks early. since that is indeed a dev-only
> > concern
> > >> it
> > >> > wont ever get used in production.
> > >> > 2. i said this on some other kip discussion: i think the nice thing
> > about
> > >> > the pool API is it "scales" from just keeping a memory bound to
> > actually
> > >> > re-using buffers without changing the calling code. i think
> > >> actuallypooling
> > >> > large buffers will result in a significant performance impact, but
> > thats
> > >> > outside the scope of this kip. at that point i think more pool
> > >> > implementations (that actually pool) would be written. i agree with
> > the
> > >> > ideal of exposing as few knobs as possible, but switching pools (or
> > pool
> > >> > params) for tuning may happen at some later point.
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Nov 11, 2016 at 

[jira] [Commented] (KAFKA-4414) Unexpected "Halting because log truncation is not allowed"

[Meyer Kizner (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670636#comment-15670636
 ] 

Meyer Kizner commented on KAFKA-4414:
-

What value would you suggest? We're already using 5000ms, which I thought was 
relatively short. A shorter timeout makes the issue less likely, but it looks 
like there's a race condition here.

> Unexpected "Halting because log truncation is not allowed"
> --
>
> Key: KAFKA-4414
> URL: https://issues.apache.org/jira/browse/KAFKA-4414
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Meyer Kizner
>
> Our Kafka installation runs with unclean leader election disabled, so brokers 
> halt when they find that their message offset is ahead of the leader's offset 
> for a topic. We had two brokers halt today with this issue. After much time 
> spent digging through the logs, I believe the following timeline describes 
> what occurred and points to a plausible hypothesis as to what happened.
> * B1, B2, and B3 are replicas of a topic, all in the ISR. B2 is currently the 
> leader, but B1 is the preferred leader. The controller runs on B3.
> * B1 fails, but the controller does not detect the failure immediately.
> * B2 receives a message from a producer and B3 fetches it to stay up to date. 
> B2 has not accepted the message, because B1 is down and so has not 
> acknowledged the message.
> * The controller triggers a preferred leader election, making B1 the leader, 
> and notifies all replicas.
> * Very shortly afterwards (~200ms), B1's broker registration in ZooKeeper 
> expires, so the controller reassigns B2 to be leader again and notifies all 
> replicas.
> * Because B3 is the controller, while B2 is on another box, B3 hears about 
> both of these events before B2 hears about either. B3 truncates its log to 
> the high water mark (before the pending message) and resumes fetching from B2.
> * B3 fetches the pending message from B2 again.
> * B2 learns that it has been displaced and then reelected, and truncates its 
> log to the high water mark, before the pending message.
> * The next time B3 tries to fetch from B2, it sees that B2 is missing the 
> pending message and halts.
> In this case, there was no data loss or inconsistency. I haven't fully 
> thought through whether either would be possible, but it seems likely that 
> they would be, especially if there had been multiple producers to this topic.
> I'm not completely certain about this timeline, but this sequence of events 
> appears to at least be possible. Looking a bit through the controller code, 
> there doesn't seem to be anything that forces {{LeaderAndIsrRequest}} to be 
> sent in a particular order. If someone with more knowledge of the code base 
> believes this is incorrect, I'd be happy to post the logs and/or do some more 
> digging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670575#comment-15670575
 ] 

Rajini Sivaram commented on KAFKA-4406:
---

Perhaps it is worth writing a small KIP explaining the motivation and how the 
different uses of security providers will be handled in brokers and clients. 
[~ijuma] What do you think?

> Add support for custom Java Security Providers in configuration
> ---
>
> Key: KAFKA-4406
> URL: https://issues.apache.org/jira/browse/KAFKA-4406
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Magnus Reftel
>Priority: Minor
>
> Currently, the only way to add a custom security provider is though adding a 
> -Djava.security.properties= option to the command line, e.g. though 
> KAFKA_OPTS. It would be more convenient if this could be done though the 
> config file, like all the other SSL related options.
> I propose adding a new configuration option, ssl.provider.classes, which 
> holds a list of names of security provider classes that will be loaded, 
> instantiated, and added before creating SSL contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[Magnus Reftel (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670468#comment-15670468
 ] 

Magnus Reftel commented on KAFKA-4406:
--

Right, so `ssl.` is out. How about `security.providers`?

For SASL, another option would be that if the user specified a preference of 
provider, then get the factory class manually via 
`Class.forName(Security.getProvider(configuredName).get("SaslClientFactory." + 
configuredMechanism))`. That way, we'd be able to use provider names in all 
places where we use the JCA provider mechanism.

How about the use of MessageDigest that you pointed out (only used by 
SkimpyOffsetMap.scala it seems)? Should there be some way of selecting provider 
there as well, for consistency?

> Add support for custom Java Security Providers in configuration
> ---
>
> Key: KAFKA-4406
> URL: https://issues.apache.org/jira/browse/KAFKA-4406
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Magnus Reftel
>Priority: Minor
>
> Currently, the only way to add a custom security provider is though adding a 
> -Djava.security.properties= option to the command line, e.g. though 
> KAFKA_OPTS. It would be more convenient if this could be done though the 
> config file, like all the other SSL related options.
> I propose adding a new configuration option, ssl.provider.classes, which 
> holds a list of names of security provider classes that will be loaded, 
> instantiated, and added before creating SSL contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670416#comment-15670416
 ] 

Rajini Sivaram commented on KAFKA-4406:
---

Java security providers are used not just for cryptographic services, but for 
any security service with a pluggable architecture. This includes SASL, which 
is used by Kafka for authentication. And JCA provider is used in 
`MessageDigest` as well, used in the broker even without SSL. 

You are right that it is unlikely that anyone would want to replace an existing 
SSL provider and provider names are configurable for SSL. But if you wanted to 
use the same option to configure SASL providers, you might want to replace an 
existing provider, which is looked up by SASL mechanism.

> Add support for custom Java Security Providers in configuration
> ---
>
> Key: KAFKA-4406
> URL: https://issues.apache.org/jira/browse/KAFKA-4406
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Magnus Reftel
>Priority: Minor
>
> Currently, the only way to add a custom security provider is though adding a 
> -Djava.security.properties= option to the command line, e.g. though 
> KAFKA_OPTS. It would be more convenient if this could be done though the 
> config file, like all the other SSL related options.
> I propose adding a new configuration option, ssl.provider.classes, which 
> holds a list of names of security provider classes that will be loaded, 
> instantiated, and added before creating SSL contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[Magnus Reftel (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670080#comment-15670080
 ] 

Magnus Reftel commented on KAFKA-4406:
--

The PR adds it for both broker and clients, so both can be configured using the 
same property, if desired. The name of the property could definitely be better, 
though, as it's not tied to ssl.provider (which is used as provider name for 
the SSL context, and the SSL context only, whereas the provider classes are 
relevant to all algorithm lookups). All other settings related to security 
start with `ssl.`, though, so it should probably be in that part of the tree 
(and I find no usages of JCA provider mechanism outside of the SSL code). Any 
suggestions?

I don't quite get what the use of replacing an existing provider would be. If 
overriding the implementation of an existing algorithm, then specifying the 
provider name (like one can do for SSL contexts using the `ssl.provider` 
setting - I guess this would be useful also in the other places where one can 
specify algorithm names) is the standard way of doing it. If adding a new 
algorithm, then using the name of the new one as e.g. 
`ssl.keymanager.algorithm` would suffice. What am I missing?

> Add support for custom Java Security Providers in configuration
> ---
>
> Key: KAFKA-4406
> URL: https://issues.apache.org/jira/browse/KAFKA-4406
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Magnus Reftel
>Priority: Minor
>
> Currently, the only way to add a custom security provider is though adding a 
> -Djava.security.properties= option to the command line, e.g. though 
> KAFKA_OPTS. It would be more convenient if this could be done though the 
> config file, like all the other SSL related options.
> I propose adding a new configuration option, ssl.provider.classes, which 
> holds a list of names of security provider classes that will be loaded, 
> instantiated, and added before creating SSL contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2142: Merge pull request #1 from apache/trunk

[husthang]

GitHub user husthang opened a pull request:

https://github.com/apache/kafka/pull/2142

Merge pull request #1 from apache/trunk

update test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/husthang/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2142


commit 806fd76fd2701810b568d31f729b38bda598f681
Author: husthang <981401...@qq.com>
Date:   2016-11-16T09:14:19Z

Merge pull request #1 from apache/trunk

update




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---