date:20160303

Ashish K Singh created KAFKA-3331:
-

 Summary: Refactor TopicCommand to make it testable and add unit 
tests
 Key: KAFKA-3331
 URL: https://issues.apache.org/jira/browse/KAFKA-3331
 Project: Kafka
  Issue Type: Wish
Reporter: Ashish K Singh
Assignee: Ashish K Singh


TopicCommand has become a functionality packed, hard to read, class. Adding or 
changing it with confidence requires some unit tests around it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #418

See

[GitHub] kafka pull request: KAFKA-3247: Add option to see unbalanced parti...

2016-03-03 Thread SinghAsDev

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1011

KAFKA-3247: Add option to see unbalanced partitions via kafka-topic tool



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3247

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit f4970d39e66d98dcc531a7f1c3acfacdf001c4ee
Author: Ashish Singh 
Date:   2016-03-04T02:11:28Z

KAFKA-3247: Add option to see unbalanced partitions via kafka-topic tool




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3281) Improve message of stop scripts when no processes are running


 [ 
https://issues.apache.org/jira/browse/KAFKA-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3281:

   Resolution: Fixed
Fix Version/s: 0.10.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 971
[https://github.com/apache/kafka/pull/971]

> Improve message of stop scripts when no processes are running
> -
>
> Key: KAFKA-3281
> URL: https://issues.apache.org/jira/browse/KAFKA-3281
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Sasaki Toru
>Assignee: Sasaki Toru
>Priority: Minor
> Fix For: 0.10.0.0
>
>
> Stop scritps such as kafka-server-stop.sh log messages of kill command's 
> error when processes aren't running.
> Example(Brokers are not running):
> {code}
> $ bin/kafka-server-stop.sh 
> kill: invalid argument S
> Usage:
>  kill [options]  [...]
> Options:
>   [...]send signal to every  listed
>  -, -s, --signal 
> specify the  to be sent
>  -l, --list=[]  list all signal names, or convert one to a name
>  -L, --tablelist all signal names in a nice table
>  -h, --help display this help and exit
>  -V, --version  output version information and exit
> For more details see kill(1).
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3281) Improve message of stop scripts when no processes are running


[ 
https://issues.apache.org/jira/browse/KAFKA-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179129#comment-15179129
 ] 

ASF GitHub Bot commented on KAFKA-3281:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/971


> Improve message of stop scripts when no processes are running
> -
>
> Key: KAFKA-3281
> URL: https://issues.apache.org/jira/browse/KAFKA-3281
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Sasaki Toru
>Assignee: Sasaki Toru
>Priority: Minor
> Fix For: 0.10.0.0
>
>
> Stop scritps such as kafka-server-stop.sh log messages of kill command's 
> error when processes aren't running.
> Example(Brokers are not running):
> {code}
> $ bin/kafka-server-stop.sh 
> kill: invalid argument S
> Usage:
>  kill [options]  [...]
> Options:
>   [...]send signal to every  listed
>  -, -s, --signal 
> specify the  to be sent
>  -l, --list=[]  list all signal names, or convert one to a name
>  -L, --tablelist all signal names in a nice table
>  -h, --help display this help and exit
>  -V, --version  output version information and exit
> For more details see kill(1).
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3281: Improve message of *-server-stop.s...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/971


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3187) Make kafka-acls.sh help messages more generic.


[ 
https://issues.apache.org/jira/browse/KAFKA-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179116#comment-15179116
 ] 

ASF GitHub Bot commented on KAFKA-3187:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/892


> Make kafka-acls.sh help messages more generic.
> --
>
> Key: KAFKA-3187
> URL: https://issues.apache.org/jira/browse/KAFKA-3187
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> Make kafka-acls.sh help messages generic, agnostic of 
> {{SimpleAclsAuthorizer}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3187) Make kafka-acls.sh help messages more generic.


 [ 
https://issues.apache.org/jira/browse/KAFKA-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3187:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 892
[https://github.com/apache/kafka/pull/892]

> Make kafka-acls.sh help messages more generic.
> --
>
> Key: KAFKA-3187
> URL: https://issues.apache.org/jira/browse/KAFKA-3187
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> Make kafka-acls.sh help messages generic, agnostic of 
> {{SimpleAclsAuthorizer}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1090

See

Build failed in Jenkins: kafka-trunk-jdk8 #417

See 

Changes:

[cshapi] KAFKA-3290: fix race condition with worker task shutdown and mock

[cshapi] KAFKA-3286: Add plugin to quickly check for outdated dependencies

--
[...truncated 3608 lines...]

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testTopiPartitionSerializationCompatibility PASSED

org.apache.kafka.common.serialization.SerializationTest > testStringSerializer 
PASSED

org.apache.kafka.common.serialization.SerializationTest > testIntegerSerializer 
PASSED

org.apache.kafka.common.serialization.SerializationTest > 
testByteBufferSerializer PASSED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidators PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionDefault PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueExceptions PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionInheritance PASSED

org.apache.kafka.common.protocol.ErrorsTest > testNoneException PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueErrorCodes PASSED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionsAreNotGeneric PASSED

org.apache.kafka.common.protocol.ProtoUtilsTest > schemaVersionOutOfRange PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testArray 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testNulls 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testNullableDefault PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testDefault 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testSimple 
PASSED

org.apache.kafka.common.requests.RequestResponseTest > testSerialization PASSED

org.apache.kafka.common.requests.RequestResponseTest > fetchResponseVersionTest 
PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
produceResponseVersionTest PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
testControlledShutdownResponse PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
testRequestHeaderWithNullClientId PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.metrics.MetricsTest > testSimpleStats PASSED

org.apache.kafka.common.metrics.MetricsTest > testOldDataHasNoEffect PASSED

org.apache.kafka.common.metrics.MetricsTest > testQuotasEquality PASSED

org.apache.kafka.common.metrics.MetricsTest > testRemoveInactiveMetrics PASSED

org.apache.kafka.common.metrics.MetricsTest > testMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testRateWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testTimeWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testEventWindowing PASSED

org.apache.kafka.common.metrics.MetricsTest > testRemoveMetric PASSED

org.apache.kafka.common.metrics.MetricsTest > testBadSensorHierarchy PASSED

org.apache.kafka.common.metrics.MetricsTest > testRemoveSensor PASSED

org.apache.kafka.common.metrics.MetricsTest > testPercentiles PASSED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testQuotas PASSED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors PASSED

org.apache.kafka.common.metrics.JmxReporterTest > testJmxRegistration PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme PASSED

org.apache.kafka.common.utils.CrcTest > testUpdateInt PASSED

org.apache.kafka.common.utils.CrcTest >

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

2016-03-03 Thread Becket Qin

+1 (non-binding)

On Thu, Mar 3, 2016 at 3:39 PM, Gwen Shapira  wrote:

> +1
>
> On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:
> > Hi all,
> >
> > The new Java producer was introduced in 0.8.2.0 (released in February
> > 2015). It has become the default implementation for various tools since
> > 0.9.0.0 (released in October 2015) and it is the only implementation with
> > support for the security features introduced in 0.9.0.0.
> >
> > Given this, I think we should deprecate the old Scala producers for
> > 0.10.0.0 by adding @deprecated annotations in the code and updating the
> the
> > documentation to encourage usage of the new Java producer. This would
> give
> > our users a stronger signal regarding our plans to focus on the new Java
> > producer going forward.
> >
> > Note that this proposal is only about deprecating the old Scala
> producers as,
> > in my opinion, it is too early to do the same for the old Scala
> consumers. The
> > new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> > beta. It would be good to have a full release cycle where the new
> consumer
> > is no longer in beta before we deprecate the old consumers. We are hoping
> > to remove the beta label for the consumer for 0.10.0.0, but that's a
> > separate discussion.
> >
> > With regards to removal of the deprecated producers, the current thinking
> > is to remove all Scala clients at the same time, so it will take at least
> > two non bug-fix release cycles (it could take longer depending on users'
> > feedback).
> >
> > The feedback was mostly positive in the discuss thread although some
> points
> > were raised about deprecating the old producers before deprecating the
> old
> > consumers:
> >
> >
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
> >
> > The JIRA for tracking this is KAFKA-2982.
> >
> > The vote will run for 72 hours.
> >
> > Thanks,
> > Ismael
>

[jira] [Resolved] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

2016-03-03 Thread Thomas Graves (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves resolved KAFKA-3236.
--
Resolution: Won't Fix

Ok, closing this for now then.

> Honor Producer Configuration "block.on.buffer.full"
> ---
>
> Key: KAFKA-3236
> URL: https://issues.apache.org/jira/browse/KAFKA-3236
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>
> In Kafka-0.9, "max.block.ms" is used to control how long the following 
> methods will block.
> KafkaProducer.send() when
>* Buffer is full
>* Metadata is unavailable
> KafkaProducer.partitionsFor() when
>* Metadata is unavailable
> However when "block.on.buffer.full" is set to false, "max.block.ms" is in 
> effect whenever a buffer is requested/allocated from the Producer BufferPool. 
> Instead it should throw a BufferExhaustedException without waiting for 
> "max.block.ms"
> This is particulary useful if a producer application does not wish to block 
> at all on KafkaProducer.send() . We avoid waiting on KafkaProducer.send() 
> when metadata is unavailable by invoking send() only if the producer instance 
> has fetched the metadata for the topic in a different thread using the same 
> producer instance. However "max.block.ms" is still required to specify a 
> timeout for bootstrapping the metadata fetch.
> We should resolve this limitation by decoupling "max.block.ms" and 
> "block.on.buffer.full".
>* "max.block.ms" will be used exclusively for fetching metadata when
> "block.on.buffer.full" = false (in pure non-blocking mode )
>* "max.block.ms" will be applicable to both fetching metadata as well as 
> buffer allocation when "block.on.buffer.full = true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3252: Compression type from broker confi...

2016-03-03 Thread SinghAsDev

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/1010

KAFKA-3252: Compression type from broker config should be used during log 
compaction.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-3252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1010


commit 3f447c28104a2b96b4dd2dad1709f7e04f0088ff
Author: Ashish Singh 
Date:   2016-03-02T23:52:54Z

KAFKA-3252: Compression type from broker config should be used during log 
compaction.

commit 97a436028a519e143fb70700943c08ef2a714708
Author: Ashish Singh 
Date:   2016-03-02T23:56:36Z

Fix using of convertedMessage

commit 94041c2247a234f0069f7b0426948ad203b79572
Author: Ashish Singh 
Date:   2016-03-04T00:25:05Z

Add UT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3310: Fix for NPEs observed when throttl...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/989


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3310) fetch requests can trigger repeated NPE when quota is enabled

2016-03-03 Thread Jun Rao (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-3310:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 989
[https://github.com/apache/kafka/pull/989]

> fetch requests can trigger repeated NPE when quota is enabled
> -
>
> Key: KAFKA-3310
> URL: https://issues.apache.org/jira/browse/KAFKA-3310
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0, 0.9.0.1
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> We saw the following NPE when consumer quota is enabled. NPE is triggered on 
> every fetch request from the client.
> java.lang.NullPointerException
> at 
> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:122)
> at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$3(KafkaApis.scala:419)
> at 
> kafka.server.KafkaApis$$anonfun$handleFetchRequest$1.apply(KafkaApis.scala:436)
> at 
> kafka.server.KafkaApis$$anonfun$handleFetchRequest$1.apply(KafkaApis.scala:436)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:431)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> One possible cause of this is the logic of removing inactive sensors. 
> Currently, in ClientQuotaManager, we create two sensors per clientId: a 
> throttleTimeSensor and a quotaSensor. Each sensor expires if it's not 
> actively updated for 1 hour. What can happen is that initially, the quota is 
> not exceeded. So, quotaSensor is being updated actively, but 
> throttleTimeSensor is not. At some point, throttleTimeSensor is removed by 
> the expiring thread. Now, we are in a situation that quotaSensor is 
> registered, but throttleTimeSensor is not. Later on, if the quota is 
> exceeded, we will hit the above NPE when trying to update throttleTimeSensor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3310) fetch requests can trigger repeated NPE when quota is enabled


[ 
https://issues.apache.org/jira/browse/KAFKA-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179001#comment-15179001
 ] 

ASF GitHub Bot commented on KAFKA-3310:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/989


> fetch requests can trigger repeated NPE when quota is enabled
> -
>
> Key: KAFKA-3310
> URL: https://issues.apache.org/jira/browse/KAFKA-3310
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0, 0.9.0.1
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> We saw the following NPE when consumer quota is enabled. NPE is triggered on 
> every fetch request from the client.
> java.lang.NullPointerException
> at 
> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:122)
> at 
> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$3(KafkaApis.scala:419)
> at 
> kafka.server.KafkaApis$$anonfun$handleFetchRequest$1.apply(KafkaApis.scala:436)
> at 
> kafka.server.KafkaApis$$anonfun$handleFetchRequest$1.apply(KafkaApis.scala:436)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:481)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:431)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> One possible cause of this is the logic of removing inactive sensors. 
> Currently, in ClientQuotaManager, we create two sensors per clientId: a 
> throttleTimeSensor and a quotaSensor. Each sensor expires if it's not 
> actively updated for 1 hour. What can happen is that initially, the quota is 
> not exceeded. So, quotaSensor is being updated actively, but 
> throttleTimeSensor is not. At some point, throttleTimeSensor is removed by 
> the expiring thread. Now, we are in a situation that quotaSensor is 
> registered, but throttleTimeSensor is not. Later on, if the quota is 
> exceeded, we will hit the above NPE when trying to update throttleTimeSensor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

2016-03-03 Thread Gwen Shapira

+1

On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:
> Hi all,
>
> The new Java producer was introduced in 0.8.2.0 (released in February
> 2015). It has become the default implementation for various tools since
> 0.9.0.0 (released in October 2015) and it is the only implementation with
> support for the security features introduced in 0.9.0.0.
>
> Given this, I think we should deprecate the old Scala producers for
> 0.10.0.0 by adding @deprecated annotations in the code and updating the the
> documentation to encourage usage of the new Java producer. This would give
> our users a stronger signal regarding our plans to focus on the new Java
> producer going forward.
>
> Note that this proposal is only about deprecating the old Scala producers as,
> in my opinion, it is too early to do the same for the old Scala consumers. The
> new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> beta. It would be good to have a full release cycle where the new consumer
> is no longer in beta before we deprecate the old consumers. We are hoping
> to remove the beta label for the consumer for 0.10.0.0, but that's a
> separate discussion.
>
> With regards to removal of the deprecated producers, the current thinking
> is to remove all Scala clients at the same time, so it will take at least
> two non bug-fix release cycles (it could take longer depending on users'
> feedback).
>
> The feedback was mostly positive in the discuss thread although some points
> were raised about deprecating the old producers before deprecating the old
> consumers:
>
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
>
> The JIRA for tracking this is KAFKA-2982.
>
> The vote will run for 72 hours.
>
> Thanks,
> Ismael

[jira] [Updated] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated


 [ 
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3330:
---
Fix Version/s: 0.10.0.0

> Truncate log cleaner offset checkpoint if the log is truncated
> --
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.0.0
>
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and 
> [300, 400) respectively. 300 is the base offset of the active log segment. 
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), 
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == 
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not 
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than 
> cleaner offset checkpoint, we should reset cleaner offset checkpoint to the 
> base offset of the active segment if this value is larger than the 
> checkpointed offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated


 [ 
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3330:
---
Priority: Critical  (was: Major)

> Truncate log cleaner offset checkpoint if the log is truncated
> --
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and 
> [300, 400) respectively. 300 is the base offset of the active log segment. 
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), 
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == 
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not 
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than 
> cleaner offset checkpoint, we should reset cleaner offset checkpoint to the 
> base offset of the active segment if this value is larger than the 
> checkpointed offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3286) Add plugin to quickly check for outdated dependencies


[ 
https://issues.apache.org/jira/browse/KAFKA-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178792#comment-15178792
 ] 

ASF GitHub Bot commented on KAFKA-3286:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/973


> Add plugin to quickly check for outdated dependencies
> -
>
> Key: KAFKA-3286
> URL: https://issues.apache.org/jira/browse/KAFKA-3286
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3286) Add plugin to quickly check for outdated dependencies


 [ 
https://issues.apache.org/jira/browse/KAFKA-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3286:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 973
[https://github.com/apache/kafka/pull/973]

> Add plugin to quickly check for outdated dependencies
> -
>
> Key: KAFKA-3286
> URL: https://issues.apache.org/jira/browse/KAFKA-3286
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3286: Add plugin to quickly check for ou...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/973


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated


[ 
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178784#comment-15178784
 ] 

ASF GitHub Bot commented on KAFKA-3330:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/1009

KAFKA-3330; Truncate log cleaner offset checkpoint if the log is truncated

@becketqin Can you take a look?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-3330

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1009


commit 5fc32b4020198dfeabc6bba80d4657f7de20e169
Author: Dong Lin 
Date:   2016-03-03T23:02:23Z

KAFKA-3330; Truncate log cleaner offset checkpoint if the log is truncated




> Truncate log cleaner offset checkpoint if the log is truncated
> --
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and 
> [300, 400) respectively. 300 is the base offset of the active log segment. 
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), 
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == 
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not 
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than 
> cleaner offset checkpoint, we should reset cleaner offset checkpoint to the 
> base offset of the active segment if this value is larger than the 
> checkpointed offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3330; Truncate log cleaner offset checkp...

2016-03-03 Thread lindong28

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/1009

KAFKA-3330; Truncate log cleaner offset checkpoint if the log is truncated

@becketqin Can you take a look?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-3330

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1009


commit 5fc32b4020198dfeabc6bba80d4657f7de20e169
Author: Dong Lin 
Date:   2016-03-03T23:02:23Z

KAFKA-3330; Truncate log cleaner offset checkpoint if the log is truncated




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated

2016-03-03 Thread Dong Lin (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-3330:

Summary: Truncate log cleaner offset checkpoint if the log is truncated  
(was: Decrease log cleaner offset checkpoint if the log is truncated)

> Truncate log cleaner offset checkpoint if the log is truncated
> --
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and 
> [300, 400) respectively. 300 is the base offset of the active log segment. 
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), 
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == 
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not 
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than 
> cleaner offset checkpoint, we should reset cleaner offset checkpoint to the 
> base offset of the active segment if this value is larger than the 
> checkpointed offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

2016-03-03 Thread Joe Stein

+1

~ Joestein
On Mar 3, 2016 3:01 PM, "Neha Narkhede"  wrote:

> +1
>
> On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > The new Java producer was introduced in 0.8.2.0 (released in February
> > 2015). It has become the default implementation for various tools since
> > 0.9.0.0 (released in October 2015) and it is the only implementation with
> > support for the security features introduced in 0.9.0.0.
> >
> > Given this, I think we should deprecate the old Scala producers for
> > 0.10.0.0 by adding @deprecated annotations in the code and updating the
> the
> > documentation to encourage usage of the new Java producer. This would
> give
> > our users a stronger signal regarding our plans to focus on the new Java
> > producer going forward.
> >
> > Note that this proposal is only about deprecating the old Scala producers
> > as,
> > in my opinion, it is too early to do the same for the old Scala
> consumers.
> > The
> > new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> > beta. It would be good to have a full release cycle where the new
> consumer
> > is no longer in beta before we deprecate the old consumers. We are hoping
> > to remove the beta label for the consumer for 0.10.0.0, but that's a
> > separate discussion.
> >
> > With regards to removal of the deprecated producers, the current thinking
> > is to remove all Scala clients at the same time, so it will take at least
> > two non bug-fix release cycles (it could take longer depending on users'
> > feedback).
> >
> > The feedback was mostly positive in the discuss thread although some
> points
> > were raised about deprecating the old producers before deprecating the
> old
> > consumers:
> >
> >
> >
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
> >
> > The JIRA for tracking this is KAFKA-2982.
> >
> > The vote will run for 72 hours.
> >
> > Thanks,
> > Ismael
> >
>
>
>
> --
> Thanks,
> Neha
>

Re: [VOTE] Deprecating the old Scala producers for 0.10.0.0

2016-03-03 Thread Neha Narkhede

+1

On Thu, Mar 3, 2016 at 2:36 PM, Ismael Juma  wrote:

> Hi all,
>
> The new Java producer was introduced in 0.8.2.0 (released in February
> 2015). It has become the default implementation for various tools since
> 0.9.0.0 (released in October 2015) and it is the only implementation with
> support for the security features introduced in 0.9.0.0.
>
> Given this, I think we should deprecate the old Scala producers for
> 0.10.0.0 by adding @deprecated annotations in the code and updating the the
> documentation to encourage usage of the new Java producer. This would give
> our users a stronger signal regarding our plans to focus on the new Java
> producer going forward.
>
> Note that this proposal is only about deprecating the old Scala producers
> as,
> in my opinion, it is too early to do the same for the old Scala consumers.
> The
> new Java consumer was only introduced in 0.9.0.0 and it's still marked as
> beta. It would be good to have a full release cycle where the new consumer
> is no longer in beta before we deprecate the old consumers. We are hoping
> to remove the beta label for the consumer for 0.10.0.0, but that's a
> separate discussion.
>
> With regards to removal of the deprecated producers, the current thinking
> is to remove all Scala clients at the same time, so it will take at least
> two non bug-fix release cycles (it could take longer depending on users'
> feedback).
>
> The feedback was mostly positive in the discuss thread although some points
> were raised about deprecating the old producers before deprecating the old
> consumers:
>
>
> http://search-hadoop.com/m/uyzND1KVJJmcbgAf2=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release
>
> The JIRA for tracking this is KAFKA-2982.
>
> The vote will run for 72 hours.
>
> Thanks,
> Ismael
>



-- 
Thanks,
Neha

[jira] [Updated] (KAFKA-3330) Decrease log cleaner offset checkpoint if the log is truncated

2016-03-03 Thread Dong Lin (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-3330:

Description: 
Were getting a number of failures of the log compaction thread with the
following error:
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
__consumer_offsets-93.
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
__consumer_offsets-93...
2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
__consumer_offsets-93 for 2 segments in offset range [11951210572,
11952632314).
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
due to
java.lang.IllegalArgumentException: requirement failed: Last clean offset
is 11951210572 but segment base offset is 11950300163 for log
__consumer_offsets-93.
at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.Cleaner.clean(LogCleaner.scala:306)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
~[kafka_2.10-0.8.2.56.jar:?]
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped


We found that this may be caused in the following scenario:

- we have three log segments with offset range [100, 200), [200, 300), and 
[300, 400) respectively. 300 is the base offset of the active log segment. Log 
cleaner offset checkpoint is also 300.

- After log is truncated to offset 220, the log segments become [100, 200), 
[200, 220). The Log cleaner offset checkpoint is still 300.

- After new messages are appended to the log, the log segments become [100, 
200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.

- Log cleaner cleans the log starting at offset 300. The require(offset == 
start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the 
base offset of any segments.


To fix the problem, when the log is truncated to an offset smaller than cleaner 
offset checkpoint, we should reset cleaner offset checkpoint to the base offset 
of the active segment if this value is larger than the checkpointed offset.




> Decrease log cleaner offset checkpoint if the log is truncated
> --
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and 
> [300, 400) respectively. 300 is the base offset of the active log segment. 
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), 
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == 
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not 
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than

[jira] [Commented] (KAFKA-3290) WorkerSourceTask testCommit transient failure


[ 
https://issues.apache.org/jira/browse/KAFKA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178780#comment-15178780
 ] 

ASF GitHub Bot commented on KAFKA-3290:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1008


> WorkerSourceTask testCommit transient failure
> -
>
> Key: KAFKA-3290
> URL: https://issues.apache.org/jira/browse/KAFKA-3290
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> From recent failed build:
> {code}
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testCommit FAILED
> java.lang.AssertionError:
>   Expectation failure on verify:
> Listener.onStartup(job-0): expected: 1, actual: 1
> Listener.onShutdown(job-0): expected: 1, actual: 1
> at org.easymock.internal.MocksControl.verify(MocksControl.java:225)
> at 
> org.powermock.api.easymock.internal.invocationcontrol.EasyMockMethodInvocationControl.verify(EasyMockMethodInvocationControl.java:132)
> at org.powermock.api.easymock.PowerMock.verify(PowerMock.java:1466)
> at org.powermock.api.easymock.PowerMock.verifyAll(PowerMock.java:1405)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest.testCommit(WorkerSourceTaskTest.java:221)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3290: fix race condition with worker tas...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1008


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3290: fix race condition with worker tas...

2016-03-03 Thread hachikuji

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1008

KAFKA-3290: fix race condition with worker task shutdown and mock validation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3290-REVISITED

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1008


commit a7b18259caff0b52a8371d02399a328de732feec
Author: Jason Gustafson 
Date:   2016-03-03T22:37:39Z

KAFKA-3290: fix race condition with task shutdown and mock validation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3290) WorkerSourceTask testCommit transient failure


[ 
https://issues.apache.org/jira/browse/KAFKA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178752#comment-15178752
 ] 

ASF GitHub Bot commented on KAFKA-3290:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1008

KAFKA-3290: fix race condition with worker task shutdown and mock validation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3290-REVISITED

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1008


commit a7b18259caff0b52a8371d02399a328de732feec
Author: Jason Gustafson 
Date:   2016-03-03T22:37:39Z

KAFKA-3290: fix race condition with task shutdown and mock validation




> WorkerSourceTask testCommit transient failure
> -
>
> Key: KAFKA-3290
> URL: https://issues.apache.org/jira/browse/KAFKA-3290
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> From recent failed build:
> {code}
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testCommit FAILED
> java.lang.AssertionError:
>   Expectation failure on verify:
> Listener.onStartup(job-0): expected: 1, actual: 1
> Listener.onShutdown(job-0): expected: 1, actual: 1
> at org.easymock.internal.MocksControl.verify(MocksControl.java:225)
> at 
> org.powermock.api.easymock.internal.invocationcontrol.EasyMockMethodInvocationControl.verify(EasyMockMethodInvocationControl.java:132)
> at org.powermock.api.easymock.PowerMock.verify(PowerMock.java:1466)
> at org.powermock.api.easymock.PowerMock.verifyAll(PowerMock.java:1405)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest.testCommit(WorkerSourceTaskTest.java:221)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[VOTE] Deprecating the old Scala producers for 0.10.0.0

2016-03-03 Thread Ismael Juma

Hi all,

The new Java producer was introduced in 0.8.2.0 (released in February
2015). It has become the default implementation for various tools since
0.9.0.0 (released in October 2015) and it is the only implementation with
support for the security features introduced in 0.9.0.0.

Given this, I think we should deprecate the old Scala producers for
0.10.0.0 by adding @deprecated annotations in the code and updating the the
documentation to encourage usage of the new Java producer. This would give
our users a stronger signal regarding our plans to focus on the new Java
producer going forward.

Note that this proposal is only about deprecating the old Scala producers as,
in my opinion, it is too early to do the same for the old Scala consumers. The
new Java consumer was only introduced in 0.9.0.0 and it's still marked as
beta. It would be good to have a full release cycle where the new consumer
is no longer in beta before we deprecate the old consumers. We are hoping
to remove the beta label for the consumer for 0.10.0.0, but that's a
separate discussion.

With regards to removal of the deprecated producers, the current thinking
is to remove all Scala clients at the same time, so it will take at least
two non bug-fix release cycles (it could take longer depending on users'
feedback).

The feedback was mostly positive in the discuss thread although some points
were raised about deprecating the old producers before deprecating the old
consumers:

http://search-hadoop.com/m/uyzND1KVJJmcbgAf2=+DISCUSS+Deprecating+the+old+Scala+producers+for+the+next+release

The JIRA for tracking this is KAFKA-2982.

The vote will run for 72 hours.

Thanks,
Ismael

[jira] [Created] (KAFKA-3330) Decrease log cleaner offset checkpoint if the log is truncated

2016-03-03 Thread Dong Lin (JIRA)

Dong Lin created KAFKA-3330:
---

 Summary: Decrease log cleaner offset checkpoint if the log is 
truncated
 Key: KAFKA-3330
 URL: https://issues.apache.org/jira/browse/KAFKA-3330
 Project: Kafka
  Issue Type: Bug
Reporter: Dong Lin
Assignee: Dong Lin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #416

See 

Changes:

[junrao] HOTFIX: Include RocksDB dependency in release tarballs

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-us1 (Ubuntu ubuntu ubuntu-us) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision c8b6c18f68b9efbbd5e349885d301c47baabcd98 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c8b6c18f68b9efbbd5e349885d301c47baabcd98
 > git rev-list ae1fd085873c5119609df5120c02763c5ae48142 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson1178054752043398278.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 11.175 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson588441255663541346.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.11/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJavawarning: [options] bootstrap class path 
not set in conjunction with -source 1.7
Note: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long =

[jira] [Reopened] (KAFKA-3290) WorkerSourceTask testCommit transient failure

2016-03-03 Thread Jason Gustafson (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reopened KAFKA-3290:


Looks like this is still failing: 
https://jenkins.confluent.io/job/kafka-trunk/576/console

> WorkerSourceTask testCommit transient failure
> -
>
> Key: KAFKA-3290
> URL: https://issues.apache.org/jira/browse/KAFKA-3290
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.0.0
>
>
> From recent failed build:
> {code}
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testCommit FAILED
> java.lang.AssertionError:
>   Expectation failure on verify:
> Listener.onStartup(job-0): expected: 1, actual: 1
> Listener.onShutdown(job-0): expected: 1, actual: 1
> at org.easymock.internal.MocksControl.verify(MocksControl.java:225)
> at 
> org.powermock.api.easymock.internal.invocationcontrol.EasyMockMethodInvocationControl.verify(EasyMockMethodInvocationControl.java:132)
> at org.powermock.api.easymock.PowerMock.verify(PowerMock.java:1466)
> at org.powermock.api.easymock.PowerMock.verifyAll(PowerMock.java:1405)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTaskTest.testCommit(WorkerSourceTaskTest.java:221)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: HOTFIX: Include RocksDB dependency in release ...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1007


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: HOTFIX: Include RocksDB dependency in release ...

2016-03-03 Thread miguno

GitHub user miguno opened a pull request:

https://github.com/apache/kafka/pull/1007

HOTFIX: Include RocksDB dependency in release tarballs

Without this change `./gradlew releaseTarGz` (and its variants) will not 
include the RocksDB jar in Kafka's `libs/` folder.

@guozhangwang @junrao : please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miguno/kafka trunk-rocksdb-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1007


commit 7b289349e7d24c7ab028233833f3718945d14a97
Author: Michael G. Noll 
Date:   2016-03-03T21:35:09Z

Include RocksDB dependency in release tarballs

Without this change `./gradlew releaseTarGz` (and its variants) will not
include the RocksDB jar in Kafka's `libs/` folder.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #415

See

Build failed in Jenkins: kafka-trunk-jdk7 #1088

See 

Changes:

[cshapi] KAFKA-3314: Add CDDL license to LICENSE and NOTICE file

[cshapi] MINOR: Fix typos in docs

[cshapi] KAFKA-3325: Out of date instructions in quickstart guide

--
[...truncated 1441 lines...]
kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[13] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[14] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.producer.SyncProducerTest > testNotEnoughReplicas PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.producer.SyncProducerTest > testProduceCorrectlyReceivesResponse PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.ProducerTest > testSendToNewTopic PASSED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout PASSED

kafka.producer.ProducerTest > testSendNullMessage PASSED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.tools.ConsoleConsumerTest >

[jira] [Commented] (KAFKA-2636) Producer connectivity obscured connection failure logging


[ 
https://issues.apache.org/jira/browse/KAFKA-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178642#comment-15178642
 ] 

Gwen Shapira commented on KAFKA-2636:
-

async send doesn't throw an exception (because there's nowhere to throw to), 
but if you have a callback, it will get called with the exception.


> Producer connectivity obscured connection failure logging
> -
>
> Key: KAFKA-2636
> URL: https://issues.apache.org/jira/browse/KAFKA-2636
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1, 0.8.2.2
> Environment: Windows 8 running java implementation of Kafka Producer
>Reporter: Jason Kania
>
> The Kafka Producer does not generate a visible exception when a connection 
> cannot be made. Instead DEBUG settings are required to observe the problem as 
> shown below:
> [2015-10-12 21:23:20,335] DEBUG Error connecting to node 0 at 
> 482f4769eed1:9092: (org.apache.kafka.clients.NetworkClient)
> java.io.IOException: Can't resolve address: 482f4769eed1:9092
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:138)
>   at 
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:417)
>   at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:116)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:165)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
>   at java.lang.Thread.run(Unknown Source)
> Caused by: java.nio.channels.UnresolvedAddressException
>   at sun.nio.ch.Net.checkAddress(Unknown Source)
>   at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:135)
>   ... 5 more
> [2015-10-12 21:23:20,358] DEBUG Initiating connection to node 0 at 
> 482f4769eed1:9092. (org.apache.kafka.clients.NetworkClient)
> Secondly, the errors do not identify the node by IP address making error 
> investigation more difficult especially when learning to use Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3328) SimpleAclAuthorizer can lose ACLs with frequent add/remove calls


[ 
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178633#comment-15178633
 ] 

ASF GitHub Bot commented on KAFKA-3328:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/1006

KAFKA-3328: SimpleAclAuthorizer can lose ACLs with frequent add/remov…

…e calls

Changes the SimpleAclAuthorizer to:
- Always read state from Zookeeper before updating acls
- Update local cache when modifying acls

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka simple-authorizer-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1006


commit 3483b759e1cf5e5a42ad3206eca41e8e75906b41
Author: Grant Henke 
Date:   2016-03-03T20:59:03Z

KAFKA-3328: SimpleAclAuthorizer can lose ACLs with frequent add/remove calls

Changes the SimpleAclAuthorizer to:
- Always read state from Zookeeper before updating acls
- Update local cache when modifying acls




> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> 
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the 
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate 
> their cache and get the latest value. However that takes some time. In the 
> time between the ACL change and the cache update, a new add or remove request 
> could be made. This will follow the steps listed above, and if the cache is 
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time 
> you update zookeeper any time a change is made. However, because there can be 
> multiple instances of the Authorizer, a request could come to a separate 
> authorizer and overwrite the Zookeeper state again loosing changes from 
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write 
> state from zookeeper (instead of the cache) for add/remove requests and only 
> leverage the cache for get/authorize requests. Or it could block until all 
> the live instances have updated their cache. 
> Below is a log from a failed test in the WIP [pull 
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows 
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)

[GitHub] kafka pull request: KAFKA-3328: SimpleAclAuthorizer can lose ACLs ...

2016-03-03 Thread granthenke

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/1006

KAFKA-3328: SimpleAclAuthorizer can lose ACLs with frequent add/removâ¦

â¦e calls

Changes the SimpleAclAuthorizer to:
- Always read state from Zookeeper before updating acls
- Update local cache when modifying acls

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka simple-authorizer-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1006


commit 3483b759e1cf5e5a42ad3206eca41e8e75906b41
Author: Grant Henke 
Date:   2016-03-03T20:59:03Z

KAFKA-3328: SimpleAclAuthorizer can lose ACLs with frequent add/remove calls

Changes the SimpleAclAuthorizer to:
- Always read state from Zookeeper before updating acls
- Update local cache when modifying acls




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3329) Validation script to test expected behavior of Authorizer implementations

2016-03-03 Thread Grant Henke (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178613#comment-15178613
 ] 

Grant Henke commented on KAFKA-3329:


[~singhashish] Sure ducktape tests would work and could likely help test more. 
You are right that you would need some way to define the jar and authorizer 
class you are testing. 

> Validation script to test expected behavior of Authorizer implementations
> -
>
> Key: KAFKA-3329
> URL: https://issues.apache.org/jira/browse/KAFKA-3329
> Project: Kafka
>  Issue Type: Wish
>Reporter: Grant Henke
>
> The authorizer interface and documentation defines some of the expected 
> behavior of an Authorizer implementation. However, having real tests for a 
> user implementing their own authorizer would be useful. A script like:
> {code}
> kafka-validate-authorizer.sh --authorizer-class ...
> {code}
> could be used to validate:
> * Expected operation inheritance
> ** Example: READ or WRITE automatically grants DESCRIBE
> * Expected exceptions or handling of edge cases
> ** When I add the same ACL twice
> ** When I remove an ACL that is not set
> ** When both Deny and Allow are set?
> ** When no Acl is attached to a resource?
> * Expected support for concurrent requests against multiple instances
> These same tests could be part of the Authorizer integration tests for 
> Kafka's SimpleAuthorizer implementation. 
> Users would not be required to follow all of the "default" expectations. But 
> they would at least know what assumptions their implementation breaks. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3328) SimpleAclAuthorizer can lose ACLs with frequent add/remove calls


[ 
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178516#comment-15178516
 ] 

Ashish K Singh commented on KAFKA-3328:
---

[~parth.brahmbhatt] might have some thoughts as well.

> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> 
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the 
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate 
> their cache and get the latest value. However that takes some time. In the 
> time between the ACL change and the cache update, a new add or remove request 
> could be made. This will follow the steps listed above, and if the cache is 
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time 
> you update zookeeper any time a change is made. However, because there can be 
> multiple instances of the Authorizer, a request could come to a separate 
> authorizer and overwrite the Zookeeper state again loosing changes from 
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write 
> state from zookeeper (instead of the cache) for add/remove requests and only 
> leverage the cache for get/authorize requests. Or it could block until all 
> the live instances have updated their cache. 
> Below is a log from a failed test in the WIP [pull 
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows 
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3328) SimpleAclAuthorizer can lose ACLs with frequent add/remove calls


[ 
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178513#comment-15178513
 ] 

Ashish K Singh commented on KAFKA-3328:
---

Sounds good, however can we do lazy writes to ZK? Basically, batch multiple 
acls CRUD in a single ZK write? Depending on use-case, it will be helpful.

> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> 
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the 
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate 
> their cache and get the latest value. However that takes some time. In the 
> time between the ACL change and the cache update, a new add or remove request 
> could be made. This will follow the steps listed above, and if the cache is 
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time 
> you update zookeeper any time a change is made. However, because there can be 
> multiple instances of the Authorizer, a request could come to a separate 
> authorizer and overwrite the Zookeeper state again loosing changes from 
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write 
> state from zookeeper (instead of the cache) for add/remove requests and only 
> leverage the cache for get/authorize requests. Or it could block until all 
> the live instances have updated their cache. 
> Below is a log from a failed test in the WIP [pull 
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows 
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3328) SimpleAclAuthorizer can lose ACLs with frequent add/remove calls

2016-03-03 Thread Grant Henke (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178501#comment-15178501
 ] 

Grant Henke commented on KAFKA-3328:


I think I will work on this as it is affecting my KIP-4 work. Any thoughts on 
only using the cache for read operations and always communicating via Zookeeper 
for write operations? 

> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> 
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the 
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate 
> their cache and get the latest value. However that takes some time. In the 
> time between the ACL change and the cache update, a new add or remove request 
> could be made. This will follow the steps listed above, and if the cache is 
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time 
> you update zookeeper any time a change is made. However, because there can be 
> multiple instances of the Authorizer, a request could come to a separate 
> authorizer and overwrite the Zookeeper state again loosing changes from 
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write 
> state from zookeeper (instead of the cache) for add/remove requests and only 
> leverage the cache for get/authorize requests. Or it could block until all 
> the live instances have updated their cache. 
> Below is a log from a failed test in the WIP [pull 
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows 
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3328) SimpleAclAuthorizer can lose ACLs with frequent add/remove calls

2016-03-03 Thread Grant Henke (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke reassigned KAFKA-3328:
--

Assignee: Grant Henke

> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> 
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
>  Issue Type: Bug
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the 
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate 
> their cache and get the latest value. However that takes some time. In the 
> time between the ACL change and the cache update, a new add or remove request 
> could be made. This will follow the steps listed above, and if the cache is 
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time 
> you update zookeeper any time a change is made. However, because there can be 
> multiple instances of the Authorizer, a request could come to a separate 
> authorizer and overwrite the Zookeeper state again loosing changes from 
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write 
> state from zookeeper (instead of the cache) for add/remove requests and only 
> leverage the cache for get/authorize requests. Or it could block until all 
> the live instances have updated their cache. 
> Below is a log from a failed test in the WIP [pull 
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows 
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster 
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny 
> permission for operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for 
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for 
> operations: Describe from hosts: *) 
> (kafka.security.auth.SimpleAclAuthorizer:52)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3325) Out of date instructions in quickstart guide


[ 
https://issues.apache.org/jira/browse/KAFKA-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178500#comment-15178500
 ] 

ASF GitHub Bot commented on KAFKA-3325:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1002


> Out of date instructions in quickstart guide
> 
>
> Key: KAFKA-3325
> URL: https://issues.apache.org/jira/browse/KAFKA-3325
> Project: Kafka
>  Issue Type: Bug
>Reporter: Duncan Sands
>Priority: Trivial
> Fix For: 0.10.0.0
>
>
> In http://kafka.apache.org/documentation.html#quickstart there is the 
> following:
>   Now edit these new files and set the following properties:
>   config/server-1.properties:
>   broker.id=1
>   port=9093
>   log.dir=/tmp/kafka-logs-1
>   config/server-2.properties:
>   broker.id=2
>   port=9094
>   log.dir=/tmp/kafka-logs-2
> However the latest version of these config files has the following line which 
> also needs to be adjusted otherwise you get an exception at broker startup 
> (address in use):
>   listeners=PLAINTEXT://:9092
> So I suggest changing the instructions to:
>   Now edit these new files and set the following properties:
>   config/server-1.properties:
>   broker.id=1
>   listeners=PLAINTEXT://:9093
>   port=9093
>   log.dir=/tmp/kafka-logs-1
>   config/server-2.properties:
>   broker.id=2
>   listeners=PLAINTEXT://:9094
>   port=9094
>   log.dir=/tmp/kafka-logs-2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3325) Out of date instructions in quickstart guide


 [ 
https://issues.apache.org/jira/browse/KAFKA-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3325.
-
   Resolution: Fixed
Fix Version/s: 0.10.0.0

Issue resolved by pull request 1002
[https://github.com/apache/kafka/pull/1002]

> Out of date instructions in quickstart guide
> 
>
> Key: KAFKA-3325
> URL: https://issues.apache.org/jira/browse/KAFKA-3325
> Project: Kafka
>  Issue Type: Bug
>Reporter: Duncan Sands
>Priority: Trivial
> Fix For: 0.10.0.0
>
>
> In http://kafka.apache.org/documentation.html#quickstart there is the 
> following:
>   Now edit these new files and set the following properties:
>   config/server-1.properties:
>   broker.id=1
>   port=9093
>   log.dir=/tmp/kafka-logs-1
>   config/server-2.properties:
>   broker.id=2
>   port=9094
>   log.dir=/tmp/kafka-logs-2
> However the latest version of these config files has the following line which 
> also needs to be adjusted otherwise you get an exception at broker startup 
> (address in use):
>   listeners=PLAINTEXT://:9092
> So I suggest changing the instructions to:
>   Now edit these new files and set the following properties:
>   config/server-1.properties:
>   broker.id=1
>   listeners=PLAINTEXT://:9093
>   port=9093
>   log.dir=/tmp/kafka-logs-1
>   config/server-2.properties:
>   broker.id=2
>   listeners=PLAINTEXT://:9094
>   port=9094
>   log.dir=/tmp/kafka-logs-2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3325: Out of date instructions in quicks...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1002


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Fix typos in docs

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3314) Add CDDL license to LICENSE and NOTICE file


[ 
https://issues.apache.org/jira/browse/KAFKA-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178481#comment-15178481
 ] 

ASF GitHub Bot commented on KAFKA-3314:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/997


> Add CDDL license to LICENSE and NOTICE file
> ---
>
> Key: KAFKA-3314
> URL: https://issues.apache.org/jira/browse/KAFKA-3314
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Kafka now has a binary dependency on jersey, which is dual licensed under 
> CDDL. According to http://www.apache.org/legal/resolved.html#category-a , we 
> need to add CDDL to our LICENSE and NOTICE file. 
> The discussion on this can be found at 
> https://mail-archives.apache.org/mod_mbox/www-legal-discuss/201602.mbox/%3ccafbh0q1u33uog1+xsntens7rzaa5f6hgujczwx03xotug1c...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3314) Add CDDL license to LICENSE and NOTICE file


 [ 
https://issues.apache.org/jira/browse/KAFKA-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3314:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 997
[https://github.com/apache/kafka/pull/997]

> Add CDDL license to LICENSE and NOTICE file
> ---
>
> Key: KAFKA-3314
> URL: https://issues.apache.org/jira/browse/KAFKA-3314
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Kafka now has a binary dependency on jersey, which is dual licensed under 
> CDDL. According to http://www.apache.org/legal/resolved.html#category-a , we 
> need to add CDDL to our LICENSE and NOTICE file. 
> The discussion on this can be found at 
> https://mail-archives.apache.org/mod_mbox/www-legal-discuss/201602.mbox/%3ccafbh0q1u33uog1+xsntens7rzaa5f6hgujczwx03xotug1c...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3314: Add CDDL license to LICENSE and NO...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/997


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3323) Negative offsets in Log Segment Index files due to Integer overflow when compaction is enabled

2016-03-03 Thread Konstantin Zadorozhny (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178479#comment-15178479
 ] 

Konstantin Zadorozhny commented on KAFKA-3323:
--

Is this issue is a duplicate of KAFKA-2024?

> Negative offsets in Log Segment Index files due to Integer overflow when 
> compaction is enabled 
> ---
>
> Key: KAFKA-3323
> URL: https://issues.apache.org/jira/browse/KAFKA-3323
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1.1
>Reporter: Michael Schiff
>Assignee: Jay Kreps
> Attachments: index_dump.txt, log_dump.txt
>
>
> Once the Offset Index has negative offset values, the binary search for 
> position lookup is broken. This causes consumers of compact topics to skip 
> large offset intervals when bootstrapping.  This has serious implications for 
> consumers of compact topics.
> {code}
>  /**
>* Append an entry for the given offset/location pair to the index. This 
> entry must have a larger offset than all subsequent entries.
>*/
>   def append(offset: Long, position: Int) {
> inLock(lock) {
>   require(!isFull, "Attempt to append to a full index (size = " + size + 
> ").")
>   if (size.get == 0 || offset > lastOffset) {
> debug("Adding index entry %d => %d to %s.".format(offset, position, 
> file.getName))
> this.mmap.putInt((offset - baseOffset).toInt)
> this.mmap.putInt(position)
> this.size.incrementAndGet()
> this.lastOffset = offset
> require(entries * 8 == mmap.position, entries + " entries but file 
> position in index is " + mmap.position + ".")
>   } else {
> throw new InvalidOffsetException("Attempt to append an offset (%d) to 
> position %d no larger than the last offset appended (%d) to %s."
>   .format(offset, entries, lastOffset, file.getAbsolutePath))
>   }
> }
>   }
> {code}
> OffsetIndex.append assumes that (offset - baseOffset) can be represented as 
> an integer without overflow. If the LogSegment is from a compacted topic, 
> this assumption may not be valid. The result is a quiet integer overflow, 
> which stores a negative value into the index.
> I believe that the issue is caused by the LogCleaner. Specifically, by the 
> groupings produced by 
> {code}
> /**
>* Group the segments in a log into groups totaling less than a given size. 
> the size is enforced separately for the log data and the index data.
>* We collect a group of such segments together into a single
>* destination segment. This prevents segment sizes from shrinking too much.
>*
>* @param segments The log segments to group
>* @param maxSize the maximum size in bytes for the total of all log data 
> in a group
>* @param maxIndexSize the maximum size in bytes for the total of all index 
> data in a group
>*
>* @return A list of grouped segments
>*/
>   private[log] def groupSegmentsBySize(segments: Iterable[LogSegment], 
> maxSize: Int, maxIndexSize: Int): List[Seq[LogSegment]]
> {code}
> Since this method is only concerned with grouping by size, without taking 
> baseOffset and groupMaxOffset into account, it will produce groups that when 
> cleaned into a single segment, have offsets that overflow. This is more 
> likely for topics with low key cardinality, but high update volume, as you 
> could wind up with very few cleaned records, but with very high offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3202) Add system test for KIP-31 and KIP-32 - Change message format version on the fly


[ 
https://issues.apache.org/jira/browse/KAFKA-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178471#comment-15178471
 ] 

Ismael Juma commented on KAFKA-3202:


[~becket_qin], do you have cycles to help with this system test? [~apovzner] 
will do the other two, but she's not sure if she will have time to do this one.

> Add system test for KIP-31 and KIP-32 - Change message format version on the 
> fly
> 
>
> Key: KAFKA-3202
> URL: https://issues.apache.org/jira/browse/KAFKA-3202
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
>Assignee: Anna Povzner
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The system test should cover the case that message format changes are made 
> when clients are producing/consuming. The message format change should not 
> cause client side issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Authorizer questions

2016-03-03 Thread Ashish Singh

Great idea. Copying my comment from the JIRA.

Do you think adding ducktape tests will work? It will probably require
ducktape to allow adding resources, like JARs, config files, etc. Geoff
Anderson
 is
it possible to do that?



On Thu, Mar 3, 2016 at 10:42 AM, Grant Henke  wrote:

> Related to these questions, I create a jira to track testing/validating
> these expectations for both the internal SimpleAuthorizer and any third
> party implemented authorizer:
>
> KAFKA-3329 : Validation
> script to test expected behavior of Authorizer implementations
>
>
>
> On Thu, Mar 3, 2016 at 11:33 AM, Grant Henke  wrote:
>
> > I am working on the List/Alter ACLs patch (
> > https://github.com/apache/kafka/pull/1005) for KIP-4 and have a few
> > questions around expectations for an Authorizer implementation:
> >
> >- What is the expected behavior when I add the same ACL twice?
> >- What is the expected behavior when I remove an ACL that is not set?
> >- What type of "permission inheritance" is an implementer of the
> >Authorizer interface supposed to follow:
> >- Example: READ or WRITE automatically grants DESCRIBE
> >- Is the Authorizer implementation expected to manage blocking/local
> >cache consistency across all instances?
> >   - Or should all requests go to 1 instance?
> >   - This is related to the bug found while working on this patch:
> >   KAFKA-3328 
> >
> > Thanks,
> > Grant
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 

Regards,
Ashish

Re: Authorizer questions

2016-03-03 Thread Ashish Singh

Grant, thanks for finding this. Below are my thoughts.

On Thu, Mar 3, 2016 at 9:33 AM, Grant Henke  wrote:

> I am working on the List/Alter ACLs patch (
> https://github.com/apache/kafka/pull/1005) for KIP-4 and have a few
> questions around expectations for an Authorizer implementation:
>
>- What is the expected behavior when I add the same ACL twice?
>
I think we should ignore it. Same as setting existing perm on a file in
unix.

>- What is the expected behavior when I remove an ACL that is not set?
>
 I think we should ignore this as well. As the end goal is met in either
case.

>- What type of "permission inheritance" is an implementer of the
>Authorizer interface supposed to follow:
>- Example: READ or WRITE automatically grants DESCRIBE
>
I am more in favor of having explicit permissions. Implementations can take
care of implied permissions, if they want to. However, I do not see a
reason why.

>- Is the Authorizer implementation expected to manage blocking/local
>cache consistency across all instances?
>
The caching only happens on implementation level, so I guess yes.

>   - Or should all requests go to 1 instance?
>
This will be a huge perf hit depending on usage pattern, I think.

>   - This is related to the bug found while working on this patch:
>   KAFKA-3328 
>
> Thanks,
> Grant
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-3329) Validation script to test expected behavior of Authorizer implementations


[ 
https://issues.apache.org/jira/browse/KAFKA-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178443#comment-15178443
 ] 

Ashish K Singh commented on KAFKA-3329:
---

Do you think adding ducktape tests will work? It will probably require ducktape 
to allow adding resources, like JARs, config files, etc. [~granders] is it 
possible to do that?

> Validation script to test expected behavior of Authorizer implementations
> -
>
> Key: KAFKA-3329
> URL: https://issues.apache.org/jira/browse/KAFKA-3329
> Project: Kafka
>  Issue Type: Wish
>Reporter: Grant Henke
>
> The authorizer interface and documentation defines some of the expected 
> behavior of an Authorizer implementation. However, having real tests for a 
> user implementing their own authorizer would be useful. A script like:
> {code}
> kafka-validate-authorizer.sh --authorizer-class ...
> {code}
> could be used to validate:
> * Expected operation inheritance
> ** Example: READ or WRITE automatically grants DESCRIBE
> * Expected exceptions or handling of edge cases
> ** When I add the same ACL twice
> ** When I remove an ACL that is not set
> ** When both Deny and Allow are set?
> ** When no Acl is attached to a resource?
> * Expected support for concurrent requests against multiple instances
> These same tests could be part of the Authorizer integration tests for 
> Kafka's SimpleAuthorizer implementation. 
> Users would not be required to follow all of the "default" expectations. But 
> they would at least know what assumptions their implementation breaks. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3304) KIP-35 - Retrieving protocol version


[ 
https://issues.apache.org/jira/browse/KAFKA-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178430#comment-15178430
 ] 

Ismael Juma commented on KAFKA-3304:


Tentatively setting fix version of 0.10.0.0 as compatibility of clients and 
third-party libraries seems to be an issue for many. This will depend on the 
KIP discussion and vote, however. And we hope to get input from Magnus too.

> KIP-35 - Retrieving protocol version
> 
>
> Key: KAFKA-3304
> URL: https://issues.apache.org/jira/browse/KAFKA-3304
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Uber JIRA to track adding of functionality to retrieve protocol versions. 
> More discussion can be found on 
> [KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3304) KIP-35 - Retrieving protocol version


 [ 
https://issues.apache.org/jira/browse/KAFKA-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3304:

Reviewer: Magnus Edenhill

> KIP-35 - Retrieving protocol version
> 
>
> Key: KAFKA-3304
> URL: https://issues.apache.org/jira/browse/KAFKA-3304
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Uber JIRA to track adding of functionality to retrieve protocol versions. 
> More discussion can be found on 
> [KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3284) Consider removing beta label in security documentation


 [ 
https://issues.apache.org/jira/browse/KAFKA-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3284:

Reviewer: Gwen Shapira

> Consider removing beta label in security documentation
> --
>
> Key: KAFKA-3284
> URL: https://issues.apache.org/jira/browse/KAFKA-3284
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> We currently state that our security support is beta. It would be good to 
> remove that for 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3304) KIP-35 - Retrieving protocol version


 [ 
https://issues.apache.org/jira/browse/KAFKA-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3304:
---
Fix Version/s: 0.10.0.0

> KIP-35 - Retrieving protocol version
> 
>
> Key: KAFKA-3304
> URL: https://issues.apache.org/jira/browse/KAFKA-3304
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.0.0
>
>
> Uber JIRA to track adding of functionality to retrieve protocol versions. 
> More discussion can be found on 
> [KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3304) KIP-35 - Retrieving protocol version


 [ 
https://issues.apache.org/jira/browse/KAFKA-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3304:
---
Priority: Critical  (was: Major)

> KIP-35 - Retrieving protocol version
> 
>
> Key: KAFKA-3304
> URL: https://issues.apache.org/jira/browse/KAFKA-3304
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Uber JIRA to track adding of functionality to retrieve protocol versions. 
> More discussion can be found on 
> [KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2982) Mark the old Scala producer and related classes as deprecated


 [ 
https://issues.apache.org/jira/browse/KAFKA-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2982:

Assignee: Grant Henke
Reviewer: Gwen Shapira

> Mark the old Scala producer and related classes as deprecated
> -
>
> Key: KAFKA-2982
> URL: https://issues.apache.org/jira/browse/KAFKA-2982
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Now that the new producer and consumer are released the old Scala producer 
> and consumer clients should be deprecated to encourage use of the new clients 
> and facilitate the removal of the old clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3006) Make collection default container type for sequences in the consumer API


 [ 
https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3006:

Assignee: Pierre-Yves Ritschard
Reviewer: Jason Gustafson

> Make collection default container type for sequences in the consumer API
> 
>
> Key: KAFKA-3006
> URL: https://issues.apache.org/jira/browse/KAFKA-3006
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Pierre-Yves Ritschard
>Assignee: Pierre-Yves Ritschard
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.0.0
>
>
> The KafkaConsumer API has some annoying inconsistencies in the usage of 
> collection types. For example, subscribe() takes a list, but subscription() 
> returns a set. Similarly for assign() and assignment(). We also have pause() 
> , seekToBeginning(), seekToEnd(), and resume() which annoyingly use a 
> variable argument array, which means you have to copy the result of 
> assignment() to an array if you want to pause all assigned partitions. We can 
> solve these issues by adding the following variants:
> {code}
> void subscribe(Collection topics);
> void subscribe(Collection topics, ConsumerRebalanceListener);
> void assign(Collection partitions);
> void pause(Collection partitions);
> void resume(Collection partitions);
> void seekToBeginning(Collection);
> void seekToEnd(Collection);
> {code}
> This issues supersedes KAFKA-2991



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()


 [ 
https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3190:

Reviewer: Guozhang Wang

> KafkaProducer should not invoke callback in send()
> --
>
> Key: KAFKA-3190
> URL: https://issues.apache.org/jira/browse/KAFKA-3190
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Currently KafkaProducer will invoke callback.onComplete() if it receives an 
> ApiException during send(). This breaks the guarantee that callback will be 
> invoked in order. It seems ApiException in send() only comes from metadata 
> refresh. If so, we can probably simply throw it instead of invoking 
> callback().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3283) Consider marking the new consumer as production-ready


 [ 
https://issues.apache.org/jira/browse/KAFKA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3283:

Assignee: Jason Gustafson
Reviewer: Gwen Shapira

> Consider marking the new consumer as production-ready
> -
>
> Key: KAFKA-3283
> URL: https://issues.apache.org/jira/browse/KAFKA-3283
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Ismael Juma
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Ideally, we would:
> * Remove the beta label
> * Remove the `Unstable` annotation
> * Filling any critical gaps in functionality
> * Update the documentation on the old consumers to recommend the new consumer 
> (without deprecating the old consumer, however)
> The new consumer already looks pretty good in 0.9.0.1 so it's feasible that 
> we may be able to do this for 0.10.0.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2982) Mark the old Scala producer and related classes as deprecated


 [ 
https://issues.apache.org/jira/browse/KAFKA-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2982:
---
Priority: Critical  (was: Major)

> Mark the old Scala producer and related classes as deprecated
> -
>
> Key: KAFKA-2982
> URL: https://issues.apache.org/jira/browse/KAFKA-2982
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Now that the new producer and consumer are released the old Scala producer 
> and consumer clients should be deprecated to encourage use of the new clients 
> and facilitate the removal of the old clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3284) Consider removing beta label in security documentation


 [ 
https://issues.apache.org/jira/browse/KAFKA-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3284:
---
Priority: Critical  (was: Major)

> Consider removing beta label in security documentation
> --
>
> Key: KAFKA-3284
> URL: https://issues.apache.org/jira/browse/KAFKA-3284
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> We currently state that our security support is beta. It would be good to 
> remove that for 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3006) Make collection default container type for sequences in the consumer API


[ 
https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178407#comment-15178407
 ] 

Ismael Juma commented on KAFKA-3006:


Marked this as critical with a fix version of 0.10.0.0. We need to decide one 
way or another before the next release as incompatible changes will much harder 
after that.

> Make collection default container type for sequences in the consumer API
> 
>
> Key: KAFKA-3006
> URL: https://issues.apache.org/jira/browse/KAFKA-3006
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Pierre-Yves Ritschard
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.0.0
>
>
> The KafkaConsumer API has some annoying inconsistencies in the usage of 
> collection types. For example, subscribe() takes a list, but subscription() 
> returns a set. Similarly for assign() and assignment(). We also have pause() 
> , seekToBeginning(), seekToEnd(), and resume() which annoyingly use a 
> variable argument array, which means you have to copy the result of 
> assignment() to an array if you want to pause all assigned partitions. We can 
> solve these issues by adding the following variants:
> {code}
> void subscribe(Collection topics);
> void subscribe(Collection topics, ConsumerRebalanceListener);
> void assign(Collection partitions);
> void pause(Collection partitions);
> void resume(Collection partitions);
> void seekToBeginning(Collection);
> void seekToEnd(Collection);
> {code}
> This issues supersedes KAFKA-2991



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1215) Rack-Aware replica assignment option


 [ 
https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-1215:

Reviewer: Jun Rao

> Rack-Aware replica assignment option
> 
>
> Key: KAFKA-1215
> URL: https://issues.apache.org/jira/browse/KAFKA-1215
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.8.0
>Reporter: Joris Van Remoortere
>Assignee: Allen Wang
>Priority: Critical
> Fix For: 0.10.0.0
>
> Attachments: rack_aware_replica_assignment_v1.patch, 
> rack_aware_replica_assignment_v2.patch
>
>
> Adding a rack-id to kafka config. This rack-id can be used during replica 
> assignment by using the max-rack-replication argument in the admin scripts 
> (create topic, etc.). By default the original replication assignment 
> algorithm is used because max-rack-replication defaults to -1. 
> max-rack-replication > -1 is not honored if you are doing manual replica 
> assignment (preffered).
> If this looks good I can add some test cases specific to the rack-aware 
> assignment.
> I can also port this to trunk. We are currently running 0.8.0 in production 
> and need this, so i wrote the patch against that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3006) Make collection default container type for sequences in the consumer API


 [ 
https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3006:
---
Priority: Critical  (was: Major)

> Make collection default container type for sequences in the consumer API
> 
>
> Key: KAFKA-3006
> URL: https://issues.apache.org/jira/browse/KAFKA-3006
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Pierre-Yves Ritschard
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.0.0
>
>
> The KafkaConsumer API has some annoying inconsistencies in the usage of 
> collection types. For example, subscribe() takes a list, but subscription() 
> returns a set. Similarly for assign() and assignment(). We also have pause() 
> , seekToBeginning(), seekToEnd(), and resume() which annoyingly use a 
> variable argument array, which means you have to copy the result of 
> assignment() to an array if you want to pause all assigned partitions. We can 
> solve these issues by adding the following variants:
> {code}
> void subscribe(Collection topics);
> void subscribe(Collection topics, ConsumerRebalanceListener);
> void assign(Collection partitions);
> void pause(Collection partitions);
> void resume(Collection partitions);
> void seekToBeginning(Collection);
> void seekToEnd(Collection);
> {code}
> This issues supersedes KAFKA-2991



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3006) Make collection default container type for sequences in the consumer API


 [ 
https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3006:
---
Fix Version/s: 0.10.0.0

> Make collection default container type for sequences in the consumer API
> 
>
> Key: KAFKA-3006
> URL: https://issues.apache.org/jira/browse/KAFKA-3006
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Pierre-Yves Ritschard
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.0.0
>
>
> The KafkaConsumer API has some annoying inconsistencies in the usage of 
> collection types. For example, subscribe() takes a list, but subscription() 
> returns a set. Similarly for assign() and assignment(). We also have pause() 
> , seekToBeginning(), seekToEnd(), and resume() which annoyingly use a 
> variable argument array, which means you have to copy the result of 
> assignment() to an array if you want to pause all assigned partitions. We can 
> solve these issues by adding the following variants:
> {code}
> void subscribe(Collection topics);
> void subscribe(Collection topics, ConsumerRebalanceListener);
> void assign(Collection partitions);
> void pause(Collection partitions);
> void resume(Collection partitions);
> void seekToBeginning(Collection);
> void seekToEnd(Collection);
> {code}
> This issues supersedes KAFKA-2991



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()


 [ 
https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3190:
---
Priority: Critical  (was: Major)

> KafkaProducer should not invoke callback in send()
> --
>
> Key: KAFKA-3190
> URL: https://issues.apache.org/jira/browse/KAFKA-3190
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Currently KafkaProducer will invoke callback.onComplete() if it receives an 
> ApiException during send(). This breaks the guarantee that callback will be 
> invoked in order. It seems ApiException in send() only comes from metadata 
> refresh. If so, we can probably simply throw it instead of invoking 
> callback().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2658) Implement SASL/PLAIN


 [ 
https://issues.apache.org/jira/browse/KAFKA-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2658:

Reviewer: Jun Rao

> Implement SASL/PLAIN
> 
>
> Key: KAFKA-2658
> URL: https://issues.apache.org/jira/browse/KAFKA-2658
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> KAFKA-1686 supports SASL/Kerberos using GSSAPI. We should enable more SASL 
> mechanisms. SASL/PLAIN would enable a simpler use of SASL, which along with 
> SSL provides a secure Kafka that uses username/password for client 
> authentication.
> SASL/PLAIN protocol and its uses are described in 
> [https://tools.ietf.org/html/rfc4616]. It is supported in Java.
> This should be implemented after KAFKA-1686. This task should also hopefully 
> enable simpler unit testing of the SASL code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3283) Consider marking the new consumer as production-ready


 [ 
https://issues.apache.org/jira/browse/KAFKA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3283:
---
Priority: Critical  (was: Major)

> Consider marking the new consumer as production-ready
> -
>
> Key: KAFKA-3283
> URL: https://issues.apache.org/jira/browse/KAFKA-3283
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Ismael Juma
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Ideally, we would:
> * Remove the beta label
> * Remove the `Unstable` annotation
> * Filling any critical gaps in functionality
> * Update the documentation on the old consumers to recommend the new consumer 
> (without deprecating the old consumer, however)
> The new consumer already looks pretty good in 0.9.0.1 so it's feasible that 
> we may be able to do this for 0.10.0.0. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3201) Add system test for KIP-31 and KIP-32 - Upgrade Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3201:

Reviewer: Ewen Cheslack-Postava

> Add system test for KIP-31 and KIP-32 - Upgrade Test
> 
>
> Key: KAFKA-3201
> URL: https://issues.apache.org/jira/browse/KAFKA-3201
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
>Assignee: Anna Povzner
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> This system test should test the procedure to upgrade a Kafka broker from 
> 0.8.x and 0.9.0 to 0.10.0
> The procedure is documented in KIP-32:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3202) Add system test for KIP-31 and KIP-32 - Change message format version on the fly


 [ 
https://issues.apache.org/jira/browse/KAFKA-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3202:

Assignee: Anna Povzner
Reviewer: Ewen Cheslack-Postava

> Add system test for KIP-31 and KIP-32 - Change message format version on the 
> fly
> 
>
> Key: KAFKA-3202
> URL: https://issues.apache.org/jira/browse/KAFKA-3202
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
>Assignee: Anna Povzner
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The system test should cover the case that message format changes are made 
> when clients are producing/consuming. The message format change should not 
> cause client side issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3188) Add system test for KIP-31 and KIP-32 - Compatibility Test


 [ 
https://issues.apache.org/jira/browse/KAFKA-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3188:

Reviewer: Ewen Cheslack-Postava

> Add system test for KIP-31 and KIP-32 - Compatibility Test
> --
>
> Key: KAFKA-3188
> URL: https://issues.apache.org/jira/browse/KAFKA-3188
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Anna Povzner
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The integration test should test the compatibility between 0.10.0 broker with 
> clients on older versions. The clients version should include 0.9.0 and 0.8.x.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3197) Producer can send message out of order even when in flight request is set to 1.


 [ 
https://issues.apache.org/jira/browse/KAFKA-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3197:

Reviewer: Jun Rao

> Producer can send message out of order even when in flight request is set to 
> 1.
> ---
>
> Key: KAFKA-3197
> URL: https://issues.apache.org/jira/browse/KAFKA-3197
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The issue we saw is following:
> 1. Producer send message 0 to topic-partition-0 on broker A. The in-flight 
> request to broker A is 1.
> 2. The request is somehow lost
> 3. Producer refreshed its topic metadata and found leader of 
> topic-partition-0 migrated from broker A to broker B.
> 4. Because there is no in-flight request to broker B. All the subsequent 
> messages to topic-partition-0 in the record accumulator are sent to broker B.
> 5. Later on when the request in step (1) times out, message 0 will be retried 
> and sent to broker B. At this point, all the later messages has already been 
> sent, so we have re-order.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3303) Pass partial record metadata to Interceptor onAcknowledgement in case of errors


 [ 
https://issues.apache.org/jira/browse/KAFKA-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3303:

Reviewer: Jun Rao

> Pass partial record metadata to Interceptor onAcknowledgement in case of 
> errors
> ---
>
> Key: KAFKA-3303
> URL: https://issues.apache.org/jira/browse/KAFKA-3303
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Currently Interceptor.onAcknowledgement behaves similarly to Callback. If 
> exception occurred and exception is passed to onAcknowledgement, metadata 
> param is set to null.
> However, it would be useful to pass topic, and partition if available to the 
> interceptor so that it knows which topic/partition got an error.
> This is part of KIP-42.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3314) Add CDDL license to LICENSE and NOTICE file


 [ 
https://issues.apache.org/jira/browse/KAFKA-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3314:

Reviewer: Gwen Shapira

> Add CDDL license to LICENSE and NOTICE file
> ---
>
> Key: KAFKA-3314
> URL: https://issues.apache.org/jira/browse/KAFKA-3314
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Jun Rao
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Kafka now has a binary dependency on jersey, which is dual licensed under 
> CDDL. According to http://www.apache.org/legal/resolved.html#category-a , we 
> need to add CDDL to our LICENSE and NOTICE file. 
> The discussion on this can be found at 
> https://mail-archives.apache.org/mod_mbox/www-legal-discuss/201602.mbox/%3ccafbh0q1u33uog1+xsntens7rzaa5f6hgujczwx03xotug1c...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2370) Add pause/unpause connector support


 [ 
https://issues.apache.org/jira/browse/KAFKA-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2370:

Reviewer: Gwen Shapira

> Add pause/unpause connector support
> ---
>
> Key: KAFKA-2370
> URL: https://issues.apache.org/jira/browse/KAFKA-2370
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> It will sometimes be useful to pause/unpause connectors. For example, if you 
> know planned maintenance will occur on the source/destination system, it 
> would make sense to pause and then resume (but not delete and then restore), 
> a connector.
> This likely requires support in all Coordinator implementations 
> (standalone/distributed) to trigger the events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3316) Add Connect REST API to list available connector classes


 [ 
https://issues.apache.org/jira/browse/KAFKA-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3316:

Reviewer: Gwen Shapira

> Add Connect REST API to list available connector classes
> 
>
> Key: KAFKA-3316
> URL: https://issues.apache.org/jira/browse/KAFKA-3316
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Each worker process's REST API should have an endpoint that can list 
> available connector classes. This can use the same Reflections code as we 
> used in KAFKA-2422 to find matching connector classes based on a short name. 
> This is useful both for debugging and for any systems that want to work with 
> different connect clusters and be able to tell which clusters support which 
> connectors.
> We may need a new top-level resource to support this. We have /connectors 
> already, but that refers to instantiated connectors that have been named. In 
> contrast, this resource would refer to the connector classes 
> (uninstantiated). We might be able to use the same resource to, e.g., lookup 
> config info in KAFKA-3315 (which occurs before connector instantiation).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1087

See

[jira] [Updated] (KAFKA-3323) Negative offsets in Log Segment Index files due to Integer overflow when compaction is enabled

2016-03-03 Thread Michael Schiff (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Schiff updated KAFKA-3323:
--
Description: 
Once the Offset Index has negative offset values, the binary search for 
position lookup is broken. This causes consumers of compact topics to skip 
large offset intervals when bootstrapping.  This has serious implications for 
consumers of compact topics.

{code}
 /**
   * Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
   */
  def append(offset: Long, position: Int) {
inLock(lock) {
  require(!isFull, "Attempt to append to a full index (size = " + size + 
").")
  if (size.get == 0 || offset > lastOffset) {
debug("Adding index entry %d => %d to %s.".format(offset, position, 
file.getName))
this.mmap.putInt((offset - baseOffset).toInt)
this.mmap.putInt(position)
this.size.incrementAndGet()
this.lastOffset = offset
require(entries * 8 == mmap.position, entries + " entries but file 
position in index is " + mmap.position + ".")
  } else {
throw new InvalidOffsetException("Attempt to append an offset (%d) to 
position %d no larger than the last offset appended (%d) to %s."
  .format(offset, entries, lastOffset, file.getAbsolutePath))
  }
}
  }
{code}

OffsetIndex.append assumes that (offset - baseOffset) can be represented as an 
integer without overflow. If the LogSegment is from a compacted topic, this 
assumption may not be valid. The result is a quiet integer overflow, which 
stores a negative value into the index.

I believe that the issue is caused by the LogCleaner. Specifically, by the 
groupings produced by 
{code}
/**
   * Group the segments in a log into groups totaling less than a given size. 
the size is enforced separately for the log data and the index data.
   * We collect a group of such segments together into a single
   * destination segment. This prevents segment sizes from shrinking too much.
   *
   * @param segments The log segments to group
   * @param maxSize the maximum size in bytes for the total of all log data in 
a group
   * @param maxIndexSize the maximum size in bytes for the total of all index 
data in a group
   *
   * @return A list of grouped segments
   */
  private[log] def groupSegmentsBySize(segments: Iterable[LogSegment], maxSize: 
Int, maxIndexSize: Int): List[Seq[LogSegment]]
{code}

Since this method is only concerned with grouping by size, without taking 
baseOffset and groupMaxOffset into account, it will produce groups that when 
cleaned into a single segment, have offsets that overflow. This is more likely 
for topics with low key cardinality, but high update volume, as you could wind 
up with very few cleaned records, but with very high offsets.

  was:
Once the Offset Index has negative offset values, the binary search for 
position lookup is broken. This causes consumers of compact topics to skip 
large offset intervals when bootstrapping.  This has serious implications for 
consumers of compact topics.

{code}
 /**
   * Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
   */
  def append(offset: Long, position: Int) {
inLock(lock) {
  require(!isFull, "Attempt to append to a full index (size = " + size + 
").")
  if (size.get == 0 || offset > lastOffset) {
debug("Adding index entry %d => %d to %s.".format(offset, position, 
file.getName))
this.mmap.putInt((offset - baseOffset).toInt)
this.mmap.putInt(position)
this.size.incrementAndGet()
this.lastOffset = offset
require(entries * 8 == mmap.position, entries + " entries but file 
position in index is " + mmap.position + ".")
  } else {
throw new InvalidOffsetException("Attempt to append an offset (%d) to 
position %d no larger than the last offset appended (%d) to %s."
  .format(offset, entries, lastOffset, file.getAbsolutePath))
  }
}
  }
{code}

OffsetIndex.append assumes that (offset - baseOffset) can be represented as an 
integer without overflow. If the LogSegment is from a compacted topic, this 
assumption may not be valid. The result is a quiet integer overflow, which 
stores a negative value into the index. This breaks the binary search used to 
lookup offset positions -> large intervals of offsets are skipped by consumers 
who are bootstrapping themselves on the topic.

I believe that the issue is caused by the LogCleaner. Specifically, by the 
groupings produced by 
{code}
/**
   * Group the segments in a log into groups totaling less than a given size. 
the size is enforced separately for the log data and the index data.
   * We collect a group of such segments together into a single
   * destination segment. This prevents segment sizes from

[jira] [Updated] (KAFKA-3323) Negative offsets in Log Segment Index files due to Integer overflow when compaction is enabled

2016-03-03 Thread Michael Schiff (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Schiff updated KAFKA-3323:
--
Description: 
Once the Offset Index has negative offset values, the binary search for 
position lookup is broken. This causes consumers of compact topics to skip 
large offset intervals when bootstrapping.  This has serious implications for 
consumers of compact topics.

{code}
 /**
   * Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
   */
  def append(offset: Long, position: Int) {
inLock(lock) {
  require(!isFull, "Attempt to append to a full index (size = " + size + 
").")
  if (size.get == 0 || offset > lastOffset) {
debug("Adding index entry %d => %d to %s.".format(offset, position, 
file.getName))
this.mmap.putInt((offset - baseOffset).toInt)
this.mmap.putInt(position)
this.size.incrementAndGet()
this.lastOffset = offset
require(entries * 8 == mmap.position, entries + " entries but file 
position in index is " + mmap.position + ".")
  } else {
throw new InvalidOffsetException("Attempt to append an offset (%d) to 
position %d no larger than the last offset appended (%d) to %s."
  .format(offset, entries, lastOffset, file.getAbsolutePath))
  }
}
  }
{code}

OffsetIndex.append assumes that (offset - baseOffset) can be represented as an 
integer without overflow. If the LogSegment is from a compacted topic, this 
assumption may not be valid. The result is a quiet integer overflow, which 
stores a negative value into the index. This breaks the binary search used to 
lookup offset positions -> large intervals of offsets are skipped by consumers 
who are bootstrapping themselves on the topic.

I believe that the issue is caused by the LogCleaner. Specifically, by the 
groupings produced by 
{code}
/**
   * Group the segments in a log into groups totaling less than a given size. 
the size is enforced separately for the log data and the index data.
   * We collect a group of such segments together into a single
   * destination segment. This prevents segment sizes from shrinking too much.
   *
   * @param segments The log segments to group
   * @param maxSize the maximum size in bytes for the total of all log data in 
a group
   * @param maxIndexSize the maximum size in bytes for the total of all index 
data in a group
   *
   * @return A list of grouped segments
   */
  private[log] def groupSegmentsBySize(segments: Iterable[LogSegment], maxSize: 
Int, maxIndexSize: Int): List[Seq[LogSegment]]
{code}

Since this method is only concerned with grouping by size, without taking 
baseOffset and groupMaxOffset into account, it will produce groups that when 
cleaned into a single segment, have offsets that overflow. This is more likely 
for topics with low key cardinality, but high update volume, as you could wind 
up with very few cleaned records, but with very high offsets.

  was:
{code}
 /**
   * Append an entry for the given offset/location pair to the index. This 
entry must have a larger offset than all subsequent entries.
   */
  def append(offset: Long, position: Int) {
inLock(lock) {
  require(!isFull, "Attempt to append to a full index (size = " + size + 
").")
  if (size.get == 0 || offset > lastOffset) {
debug("Adding index entry %d => %d to %s.".format(offset, position, 
file.getName))
this.mmap.putInt((offset - baseOffset).toInt)
this.mmap.putInt(position)
this.size.incrementAndGet()
this.lastOffset = offset
require(entries * 8 == mmap.position, entries + " entries but file 
position in index is " + mmap.position + ".")
  } else {
throw new InvalidOffsetException("Attempt to append an offset (%d) to 
position %d no larger than the last offset appended (%d) to %s."
  .format(offset, entries, lastOffset, file.getAbsolutePath))
  }
}
  }
{code}

OffsetIndex.append assumes that (offset - baseOffset) can be represented as an 
integer without overflow. If the LogSegment is from a compacted topic, this 
assumption may not be valid. The result is a quiet integer overflow, which 
stores a negative value into the index. This breaks the binary search used to 
lookup offset positions -> large intervals of offsets are skipped by consumers 
who are bootstrapping themselves on the topic.

I believe that the issue is caused by the LogCleaner. Specifically, by the 
groupings produced by 
{code}
/**
   * Group the segments in a log into groups totaling less than a given size. 
the size is enforced separately for the log data and the index data.
   * We collect a group of such segments together into a single
   * destination segment. This prevents segment sizes from shrinking too much.
   *
   * @param segments The log segments to group
   * @param maxSize the

[jira] [Updated] (KAFKA-1215) Rack-Aware replica assignment option