[GitHub] kafka pull request: KAFKA-3761: Controller has RunningAsBroker ins...

[theduderog]

GitHub user theduderog opened a pull request:

https://github.com/apache/kafka/pull/1437

KAFKA-3761: Controller has RunningAsBroker instead of RunningAsController 
state

@junrao @ijuma 

This works for me in local testing.  Do you see any issues?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/theduderog/kafka KAFKA-3761-broker-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1437


commit 6278fdec8c75a6ccc610a602c8cf27c2c1706bfb
Author: Roger Hoover 
Date:   2016-05-27T05:54:50Z

Setting RunningAsController after RunningAsBroker




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-3761) Controller has RunningAsBroker instead of RunningAsController state

[Roger Hoover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3761 started by Roger Hoover.
---
> Controller has RunningAsBroker instead of RunningAsController state
> ---
>
> Key: KAFKA-3761
> URL: https://issues.apache.org/jira/browse/KAFKA-3761
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Roger Hoover
>
> In `KafkaServer.start`, we start `KafkaController`:
> {code}
> /* start kafka controller */
> kafkaController = new KafkaController(config, zkUtils, brokerState, 
> kafkaMetricsTime, metrics, threadNamePrefix)
> kafkaController.startup()
> {code}
> Which sets the state to `RunningAsController` in 
> `KafkaController.onControllerFailover`:
> `brokerState.newState(RunningAsController)`
> And this later gets set to `RunningAsBroker`.
> This doesn't match the diagram in `BrokerStates`. [~junrao] suggested that we 
> should start the controller after we register the broker in ZK, but this 
> seems tricky as we need to controller in `KafkaApis`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303544#comment-15303544
 ] 

Ewen Cheslack-Postava commented on KAFKA-2394:
--

[~cotedm] If there are risks with DailyRollingFileAppender, I definitely think 
its good to clean it up. One thing to watch out for is that this is, in some 
sense, a public API/settings. I don't think we've been clear about 
compatibility for log settings but I suspect a large number of users still rely 
on our defaults. What do we need to do re: documentation and compatibility? 
Seems like a simple docs update could take care of most of the issues -- 
there's no good migration story for changes like this, but they probably at 
least need to be clearly documented. Might also be worth a ping on the mailing 
list about the change -- I suspect nobody will object, but for stuff like this 
its a good idea to try to make a best effort to notify people of the change so 
they have a chance to object if they really need to.

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: jin xing
>Priority: Minor
>  Labels: newbie
> Attachments: log4j.properties.patch
>
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3757) Add new client to "clients" page please

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3757.
--
Resolution: Fixed

Added to the wiki, thanks for the contribution!

> Add new client to "clients" page please
> ---
>
> Key: KAFKA-3757
> URL: https://issues.apache.org/jira/browse/KAFKA-3757
> Project: Kafka
>  Issue Type: Task
>  Components: website
>Affects Versions: 0.10.0.0
>Reporter: Vadim Chekan
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
>
> I'd like kafka4net client to be added to "clients" page:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients
> This is C# client, asynchronous, all 3 compressions supported (read and 
> write), tracks leader partition changes transparently, long time in 
> production.
> Maintainer: https://github.com/vchekan/
> License: Apache-2.0
> Repository: https://github.com/ntent-ad/kafka4net
> I've registered on the cwiki.apache.org site, but I do not have permissions 
> to edit the page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3757) Add new client to "clients" page please

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava reassigned KAFKA-3757:


Assignee: Ewen Cheslack-Postava

> Add new client to "clients" page please
> ---
>
> Key: KAFKA-3757
> URL: https://issues.apache.org/jira/browse/KAFKA-3757
> Project: Kafka
>  Issue Type: Task
>  Components: website
>Affects Versions: 0.10.0.0
>Reporter: Vadim Chekan
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
>
> I'd like kafka4net client to be added to "clients" page:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients
> This is C# client, asynchronous, all 3 compressions supported (read and 
> write), tracks leader partition changes transparently, long time in 
> production.
> Maintainer: https://github.com/vchekan/
> License: Apache-2.0
> Repository: https://github.com/ntent-ad/kafka4net
> I've registered on the cwiki.apache.org site, but I do not have permissions 
> to edit the page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303531#comment-15303531
 ] 

Guozhang Wang commented on KAFKA-3740:
--

Sounds good.

> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #654

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-3710) MemoryOffsetBackingStore creates a non-daemon thread that prevents clean shutdown

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3710.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1383
[https://github.com/apache/kafka/pull/1383]

> MemoryOffsetBackingStore creates a non-daemon thread that prevents clean 
> shutdown
> -
>
> Key: KAFKA-3710
> URL: https://issues.apache.org/jira/browse/KAFKA-3710
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Peter Davis
>Assignee: Ewen Cheslack-Postava
>  Labels: easyfix, newbie, patch
> Fix For: 0.10.1.0
>
>
> MemoryOffsetBackingStore creates an ExecutorService but 
> MemoryOffsetBackingStore.stop() fails to call executor.shutdown().  This 
> creates a zombie non-daemon thread which prevents clean shutdown when running 
> a StandaloneHerder embedded in another application.
> Note that FileOffsetBackingStore extends MemoryOffsetBackingStore so is also 
> affected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3710) MemoryOffsetBackingStore creates a non-daemon thread that prevents clean shutdown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303440#comment-15303440
 ] 

ASF GitHub Bot commented on KAFKA-3710:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1383


> MemoryOffsetBackingStore creates a non-daemon thread that prevents clean 
> shutdown
> -
>
> Key: KAFKA-3710
> URL: https://issues.apache.org/jira/browse/KAFKA-3710
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Peter Davis
>Assignee: Ewen Cheslack-Postava
>  Labels: easyfix, newbie, patch
> Fix For: 0.10.1.0
>
>
> MemoryOffsetBackingStore creates an ExecutorService but 
> MemoryOffsetBackingStore.stop() fails to call executor.shutdown().  This 
> creates a zombie non-daemon thread which prevents clean shutdown when running 
> a StandaloneHerder embedded in another application.
> Note that FileOffsetBackingStore extends MemoryOffsetBackingStore so is also 
> affected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3710: MemoryOffsetBackingStore shutdown

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1383


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[Henry Cai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303428#comment-15303428
 ] 

Henry Cai commented on KAFKA-3740:
--

Looks like you want two sets of RocksDB settings, one for K/V store, one for 
range scan store.  I think for most of the small RocksDB (size < 5GB), those 
settings (target file size, or bloom filter) probably won't matter.

To make it really flexible, you would need to provide per store rocksdb 
settings.  E.g. for A join B, if A has small-size records and B has big-size 
records you would want to set the targeFileSize smaller for RocksDB A and 
targetFileSize bigger for RocksDB B.  It seems it would have the following 
overlay config structures in StreamConfig:

rocksdb.default.target_file_size_mb=16
rocksdb.default.use_bloom_filter=true
...
rocksdb.aggregation_style.target_file_size_mb=8
...
rocksdb.join_style.target_file_size_mb=32
...
rocksdb.custom1_style.target_file_size_mb=128

System understands 'default', 'aggregation_style', 'join_style' and will choose 
different style accordingly based on the context.

Application can optionally associate a RocksDB with customer style e.g. 
custom1_style (need some API enhancement).

This might get more complicated, I think for the initial version, as long as we 
can provide rocksdb.default.property this would be a big enhancement.


> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [HEADS-UP] Modification to KIP Template

[Sriram Subramanian]

++1

Sent from my iPhone

> On May 26, 2016, at 7:24 PM, Gwen Shapira  wrote:
> 
> Hi Kafka Developers,
> 
> Just a quick heads-up that I added a new section to the KIP template: "Test
> Plans".
> I think its a good habit to think about how a feature will be tested while
> planning it. I'm talking about high-level notes on system tests, not gritty
> details.
> 
> This will apply to new KIPs, not ones in discussion/implementation phases
> (although if your KIP is under discussion and you want to add test plans,
> it will be very nice of you).
> 
> I figured we all agree that thinking a bit about tests is a good idea, so I
> added it first and started a discussion later. If you strongly object,
> please respond with strong objections. Wikis are easy to edit :)
> 
> Gwen

[HEADS-UP] Modification to KIP Template

[Gwen Shapira]

Hi Kafka Developers,

Just a quick heads-up that I added a new section to the KIP template: "Test
Plans".
I think its a good habit to think about how a feature will be tested while
planning it. I'm talking about high-level notes on system tests, not gritty
details.

This will apply to new KIPs, not ones in discussion/implementation phases
(although if your KIP is under discussion and you want to add test plans,
it will be very nice of you).

I figured we all agree that thinking a bit about tests is a good idea, so I
added it first and started a discussion later. If you strongly object,
please respond with strong objections. Wikis are easy to edit :)

Gwen

Re: KAFKA-3744: Message format to identify serializer

[Gwen Shapira]

Hi David,

Thank you for bringing this up.
I do agree that improving Kafka's message metadata is important and we are
slightly lacking on the story. We don't have built in support for message
types, or for tagging messages with things like source host or source
cluster (which were frequently requested), or many other similar metadata.

I don't think adding this information to the message format is a good idea
though.
The limited # of bits means that we are very very limited in what you
express, and your specific proposal ties Kafka to very specific formats
(Avro, Text, JSON). You are limited to 4 formats because of the bytes
limitations, but I am strongly against tying Kafka to specific formats.
First, Protobuf, Thrift and XML are very popular. Second, who knows what
will people invent tomorrow? We don't want to plan being obsolete next year.

In suggest an alternative:
Lets work together to design a community-recommended schema. It will be
implemented entirely inside the payload (keys and values), we'll leave the
specific serialization to the users, but we can add some tools to support
extracting metadata and such. Flume already has something similar (the
key-value properties in the key, with hosts, timestamps and such). We can
say something like: The first byte in the value will represent the data
type, the second will be schema-ID (if any), then key-value pairs with at
least host-name and cluster-id, etc. Kind of like how HTML has its own
headers, separate from HTTP protocol.

I can draw up a more detailed proposal (not quite a KIP, since it doesn't
change Kafka proper). if there is community interest. I remember Chris
Ricommini expressing interest, and few others (Maybe LinkedIn?) asked for
specific metadata.

Thanks,

Gwen

On Thu, May 26, 2016 at 6:29 AM, David Kay  wrote:

> All, I plan to submit a KIP to begin discussion on
> https://issues.apache.org/jira/browse/KAFKA-3744 and the associated Pull
> Request https://github.com/apache/kafka/pull/1419.  I'll notify this list
> when the KIP is submitted.
>
> Please discard my previous message containing bogus subject and links.

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303345#comment-15303345
 ] 

Guozhang Wang commented on KAFKA-3740:
--

Some thoughts on the default config values:

I think there are at least three use cases of RocksDB whose default configs 
need to be treated differentially:

1. For pure key-value store with put / get / delete, this is used for KTable 
aggregation and KStream aggregation (note that for now windowed KStream 
aggregation is using a range query, which is sub-optimal, we should really 
change it to multiple gets to avoid flushing the cache).

2. For append-only puts and range queries, used for windowed KStream joins.

3. For update puts and range queries, non-key KTable-KTable joins: we are about 
to add this support and am writing up a design proposal for it.

For example, for case 1) it should usually write-heavy, assuming we have a good 
cache hit rate on top of RocksDB, then we should consider setting smaller 
number of levels config to reduce write amplification; and for 2) and 3), we 
should turn off bloom filter by default since it does not help for range 
queries.

References:

https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
https://vimeo.com/album/2920922/video/98428203

> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3689) ERROR Processor got uncaught exception. (kafka.network.Processor)

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303301#comment-15303301
 ] 

Jun Rao commented on KAFKA-3689:


Hmm, interesting. I assume that all the errors are for the same client 
connection. While this is happening, does netstat show that connection and what 
state the connection is in (e.g. closed_wait, etc)?

> ERROR Processor got uncaught exception. (kafka.network.Processor)
> -
>
> Key: KAFKA-3689
> URL: https://issues.apache.org/jira/browse/KAFKA-3689
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
> Environment: ubuntu 14.04,
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.2)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> 3 broker cluster (all 3 servers identical -  Intel Xeon E5-2670 @2.6GHz, 
> 8cores, 16 threads 64 GB RAM & 1 TB Disk)
> Kafka Cluster is managed by 3 server ZK cluster (these servers are different 
> from Kafka broker servers). All 6 servers are connected via 10G switch. 
> Producers run from external servers.
>Reporter: Buvaneswari Ramanan
>Assignee: Jun Rao
>Priority: Minor
> Fix For: 0.10.1.0, 0.10.0.1
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> As per Ismael Juma's suggestion in email thread to us...@kafka.apache.org 
> with the same subject, I am creating this bug report.
> The following error occurs in one of the brokers in our 3 broker cluster, 
> which serves about 8000 topics. These topics are single partitioned with a 
> replication factor = 3. Each topic gets data at a low rate  – 200 bytes per 
> sec.  Leaders are balanced across the topics.
> Producers run from external servers (4 Ubuntu servers with same config as the 
> brokers), each producing to 2000 topics utilizing kafka-python library.
> This error message occurs repeatedly in one of the servers. Between the hours 
> of 10:30am and 1:30pm on 5/9/16, there were about 10 Million such 
> occurrences. This was right after a cluster restart.
> This is not the first time we got this error in this broker. In those 
> instances, error occurred hours / days after cluster restart.
> =
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count 
> for address with no connections, address: /X.Y.Z.144 (actual network address 
> masked)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count 
> for address with no connections, address: /X.Y.Z.144
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3762) Log.loadSegments() should log the message in exception

[Jun Rao (JIRA)]

Jun Rao created KAFKA-3762:
--

 Summary: Log.loadSegments() should log the message in exception
 Key: KAFKA-3762
 URL: https://issues.apache.org/jira/browse/KAFKA-3762
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.10.0.0
Reporter: Jun Rao


In Log.loadSegments(), we have the following code. It would be useful to log 
the error message in IllegalArgumentException.

if(indexFile.exists()) {
  try {
  segment.index.sanityCheck()
  } catch {
case e: java.lang.IllegalArgumentException =>
  warn("Found a corrupted index file, %s, deleting and rebuilding 
index...".format(indexFile.getAbsolutePath))
  indexFile.delete()
  segment.recover(config.maxMessageSize)
  }
}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy

[Vahid S Hashemian]

Hi Guozhang,

I was looking at the implementation of StreamsPartitionAssignor through 
its unit tests and expected to find some tests that
- verify stickiness by making at least two calls to the assign() method 
(so we check the second assign() call output preserves the assignments 
coming from the first assign() call output); or
- start off by a preset assignment, call assign() after some subscription 
change, and verify the previous assignment are preserved.
But none of the methods seem to do these. Did I overlook them, or 
stickiness is being tested in some other fashion?

Also, if there is a high-level write-up about how this assignor works 
could you please point me to it? Thanks.

Regards.
--Vahid
 



From:   Guozhang Wang 
To: "dev@kafka.apache.org" 
Date:   05/02/2016 10:34 AM
Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment Strategy



Just FYI, the StreamsPartitionAssignor in Kafka Streams are already doing
some sort of sticky partitioning mechanism. This is done through the
userData field though; i.e. all group members send their current "assigned
partitions" in their join group request, which will be grouped and send to
the leader, the leader then does best-effort for sticky-partitioning.


Guozhang

On Fri, Apr 29, 2016 at 9:48 PM, Ewen Cheslack-Postava 
wrote:

> I think I'm unclear how we leverage the
> onPartitionsRevoked/onPartitionsAssigned here in any way that's 
different
> from our normal usage -- certainly you can use them to generate a diff, 
but
> you still need to commit when partitions are revoked and that has a
> non-trivial cost. Are we just saying that you might be able to save some
> overhead, e.g. closing/reopening some other resources by doing a flush 
but
> not a close() or something? You still need to flush any output and 
commit
> offsets before returning from onPartitionsRevoked, right? Otherwise you
> couldn't guarantee clean handoff of partitions.
>
> In terms of the rebalancing, the basic requirements in the KIP seem 
sound.
> Passing previous assignment data via UserData also seems reasonable 
since
> it avoids redistributing all assignment data to all members and doesn't
> rely on the next generation leader being a member of the current
> generation. Hopefully this shouldn't be surprising since I think I
> discussed this w/ Jason before he updated the relevant wiki pages :)
>
> -Ewen
>
>
> On Mon, Apr 18, 2016 at 9:34 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > HI Jason,
> >
> > Thanks for your feedback.
> >
> > I believe your suggestion on how to take advantage of this assignor is
> > valid. We can leverage onPartitionsRevoked() and 
onPartitionsAssigned()
> > callbacks and do a comparison of assigned partitions before and after 
the
> > re-balance and do the cleanup only if there is a change (e.g., if some
> > previously assigned partition is not in the assignment).
> >
> > On your second question, a number of tests that I ran shows that the 
old
> > assignments are preserved in the current implementation; except for 
when
> > the consumer group leader is killed; in which case, a fresh assignment 
is
> > performed. This is something that needs to be fixed. I tried to use 
your
> > pointers to find out where the best place is to preserve the old
> > assignment in such circumstances but have not been able to pinpoint 
it.
> If
> > you have any suggestion on this please share. Thanks.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev@kafka.apache.org
> > Date:   04/14/2016 11:37 AM
> > Subject:Re: [DISCUSS] KIP-54 Sticky Partition Assignment 
Strategy
> >
> >
> >
> > Hi Vahid,
> >
> > Thanks for the proposal. I think one of the advantages of having 
sticky
> > assignment would be reduce the need to cleanup local partition state
> > between rebalances. Do you have any thoughts on how the user would 
take
> > advantage of this assignor in the consumer to do this? Maybe one 
approach
> > is to delay cleanup until you detect a change from the previous
> assignment
> > in the onPartitionsAssigned() callback?
> >
> > Also, can you provide some detail on how the sticky assignor works at 
the
> > group protocol level? For example, do you pass old assignments through
> the
> > "UserData" field in the consumer's JoinGroup?
> >
> > Thanks,
> > Jason
> >
> > On Thu, Apr 14, 2016 at 11:05 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi all,
> > >
> > > I have started a new KIP under
> > >
> > >
> >
> >
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy

> >
> > > The corresponding JIRA is at
> > > https://issues.apache.org/jira/browse/KAFKA-2273
> > > The corresponding PR is at https://github.com/apache/kafka/pull/1020
> > >
> > > Your feedback is much appreciated.
> > >
> > > Regards,
> > > Vahid Hashemian
> > >
> > >
> >
> >
> 

[jira] [Created] (KAFKA-3761) Controller has RunningAsBroker instead of RunningAsController state

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-3761:
--

 Summary: Controller has RunningAsBroker instead of 
RunningAsController state
 Key: KAFKA-3761
 URL: https://issues.apache.org/jira/browse/KAFKA-3761
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Roger Hoover


In `KafkaServer.start`, we start `KafkaController`:

{code}
/* start kafka controller */
kafkaController = new KafkaController(config, zkUtils, brokerState, 
kafkaMetricsTime, metrics, threadNamePrefix)
kafkaController.startup()
{code}

Which sets the state to `RunningAsController` in 
`KafkaController.onControllerFailover`:

`brokerState.newState(RunningAsController)`

And this later gets set to `RunningAsBroker`.

This doesn't match the diagram in `BrokerStates`. [~junrao] suggested that we 
should start the controller after we register the broker in ZK, but this seems 
tricky as we need to controller in `KafkaApis`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3760.

   Resolution: Fixed
 Assignee: Roger Hoover  (was: Jun Rao)
 Reviewer: Jun Rao
Fix Version/s: 0.10.1.0

Fixed via https://github.com/apache/kafka/pull/1426.

> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303170#comment-15303170
 ] 

ASF GitHub Bot commented on KAFKA-3760:
---

Github user theduderog closed the pull request at:

https://github.com/apache/kafka/pull/1436


> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Jun Rao
>Priority: Minor
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3760: Set broker state as running after ...

[theduderog]

Github user theduderog closed the pull request at:

https://github.com/apache/kafka/pull/1436


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[Roger Hoover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roger Hoover updated KAFKA-3760:

Comment: was deleted

(was: Create PR:  https://github.com/apache/kafka/pull/1436)

> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Jun Rao
>Priority: Minor
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[Roger Hoover (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303141#comment-15303141
 ] 

Roger Hoover commented on KAFKA-3760:
-

Create PR:  https://github.com/apache/kafka/pull/1436

> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Jun Rao
>Priority: Minor
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303140#comment-15303140
 ] 

ASF GitHub Bot commented on KAFKA-3760:
---

GitHub user theduderog opened a pull request:

https://github.com/apache/kafka/pull/1436

KAFKA-3760: Set broker state as running after publishing to ZooKeeper

Redoing PR #1426 with a JIRA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/theduderog/kafka broker-running-after-zk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1436


commit 54052cb17b1572b50ce770168a30c5d9cbcb278a
Author: Roger Hoover 
Date:   2016-05-25T01:37:24Z

Setting broker state as running after publishing to ZK

commit cc8ce55f874121dc4c26a63ffd6c8c7eb3a8107d
Author: Roger Hoover 
Date:   2016-05-25T01:40:51Z

Restore imports




> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Jun Rao
>Priority: Minor
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3760: Set broker state as running after ...

[theduderog]

GitHub user theduderog opened a pull request:

https://github.com/apache/kafka/pull/1436

KAFKA-3760: Set broker state as running after publishing to ZooKeeper

Redoing PR #1426 with a JIRA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/theduderog/kafka broker-running-after-zk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1436


commit 54052cb17b1572b50ce770168a30c5d9cbcb278a
Author: Roger Hoover 
Date:   2016-05-25T01:37:24Z

Setting broker state as running after publishing to ZK

commit cc8ce55f874121dc4c26a63ffd6c8c7eb3a8107d
Author: Roger Hoover 
Date:   2016-05-25T01:40:51Z

Restore imports




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[Roger Hoover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roger Hoover updated KAFKA-3760:

Summary: Set broker state as running after publishing to ZooKeeper  (was: 
Setting broker state as running after publishing to ZooKeeper)

> Set broker state as running after publishing to ZooKeeper
> -
>
> Key: KAFKA-3760
> URL: https://issues.apache.org/jira/browse/KAFKA-3760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>Assignee: Jun Rao
>Priority: Minor
>
> Currently, the broker state is set to running before it registers itself in 
> ZooKeeper. This is too early in the broker lifecycle. If clients use the 
> broker state as an indicator that the broker is ready to accept requests, 
> they will get errors. This change is to delay setting the broker state to 
> running until it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3760) Setting broker state as running after publishing to ZooKeeper

[Roger Hoover (JIRA)]

Roger Hoover created KAFKA-3760:
---

 Summary: Setting broker state as running after publishing to 
ZooKeeper
 Key: KAFKA-3760
 URL: https://issues.apache.org/jira/browse/KAFKA-3760
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.10.0.0
Reporter: Roger Hoover
Assignee: Jun Rao
Priority: Minor


Currently, the broker state is set to running before it registers itself in 
ZooKeeper. This is too early in the broker lifecycle. If clients use the broker 
state as an indicator that the broker is ready to accept requests, they will 
get errors. This change is to delay setting the broker state to running until 
it's registered in ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1316

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #653

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3723: Cannot change size of schema cache for JSON converter

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-4 (docker Ubuntu ubuntu4 ubuntu yahoo-not-h2) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 3cf2de0694cf0e276d25d8c7048a9928b41969a3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3cf2de0694cf0e276d25d8c7048a9928b41969a3
 > git rev-list d5366471d2c1a611f2241362041debc0118e5422 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson4360967715551164531.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 29.812 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5697358019448268335.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJavawarning: [options] bootstrap class path 
not set in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 

[jira] [Commented] (KAFKA-3723) Cannot change size of schema cache for JSON converter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302946#comment-15302946
 ] 

ASF GitHub Bot commented on KAFKA-3723:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1401


> Cannot change size of schema cache for JSON converter
> -
>
> Key: KAFKA-3723
> URL: https://issues.apache.org/jira/browse/KAFKA-3723
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Christian Posta
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.10.1.0
>
>
> using this worker config, value.converter.schemas.cache.size, we should be 
> able to change the size of the cache. however, because of an incorrect 
> integer cast, we cannot change it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3723) Cannot change size of schema cache for JSON converter

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3723.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1401
[https://github.com/apache/kafka/pull/1401]

> Cannot change size of schema cache for JSON converter
> -
>
> Key: KAFKA-3723
> URL: https://issues.apache.org/jira/browse/KAFKA-3723
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Christian Posta
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.10.1.0
>
>
> using this worker config, value.converter.schemas.cache.size, we should be 
> able to change the size of the cache. however, because of an incorrect 
> integer cast, we cannot change it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3723 Cannot change size of schema cache ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1401


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (KAFKA-3137) Delete tombstones in log compacted topics may never get removed.

[James Cheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Cheng closed KAFKA-3137.
--

> Delete tombstones in log compacted topics may never get removed.
> 
>
> Key: KAFKA-3137
> URL: https://issues.apache.org/jira/browse/KAFKA-3137
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>
> I spoke about this with [~junrao]. I haven't tried to reproduce this, but Jun 
> said that it looks like this is possible, so I'm filing it.
> Delete tombstones in log compacted topics are deleted after 
> delete.retention.ms (at the topic level) or log.cleaner.delete.retention.ms 
> (at the broker level).
> However, we don't have per-message timestamps (at least until KIP-32 is 
> implemented). So the timestamp of the log segment file is used as a proxy. 
> However, the modification time of the log segment changes whenever a 
> compaction run happens.
> It's possible then that if log compaction happens very frequently that 
> delete.retention.ms will never be reached. In that case, the delete 
> tombstones would stay around longer than the user expected. 
> I believe that means that log compaction would have to happen more frequently 
> than delete.retention.ms. The frequency of log compaction is some calculation 
> based on segment size, the criteria for segment roll (time or bytes), the 
> min.cleanable.dirty.ratio, as well as the amount of traffic coming into the 
> log compacted topic. So it's possible, but I'm not sure how likely.
> And I would imagine that this can't be fixed until KIP-32 is available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3137) Delete tombstones in log compacted topics may never get removed.

[James Cheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Cheng resolved KAFKA-3137.

Resolution: Invalid

> Delete tombstones in log compacted topics may never get removed.
> 
>
> Key: KAFKA-3137
> URL: https://issues.apache.org/jira/browse/KAFKA-3137
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>
> I spoke about this with [~junrao]. I haven't tried to reproduce this, but Jun 
> said that it looks like this is possible, so I'm filing it.
> Delete tombstones in log compacted topics are deleted after 
> delete.retention.ms (at the topic level) or log.cleaner.delete.retention.ms 
> (at the broker level).
> However, we don't have per-message timestamps (at least until KIP-32 is 
> implemented). So the timestamp of the log segment file is used as a proxy. 
> However, the modification time of the log segment changes whenever a 
> compaction run happens.
> It's possible then that if log compaction happens very frequently that 
> delete.retention.ms will never be reached. In that case, the delete 
> tombstones would stay around longer than the user expected. 
> I believe that means that log compaction would have to happen more frequently 
> than delete.retention.ms. The frequency of log compaction is some calculation 
> based on segment size, the criteria for segment roll (time or bytes), the 
> min.cleanable.dirty.ratio, as well as the amount of traffic coming into the 
> log compacted topic. So it's possible, but I'm not sure how likely.
> And I would imagine that this can't be fixed until KIP-32 is available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3137) Delete tombstones in log compacted topics may never get removed.

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302857#comment-15302857
 ] 

James Cheng commented on KAFKA-3137:


Jay was right about this. The compacted segments retain the modification 
timestamp of the cleaned file. Closing.

> Delete tombstones in log compacted topics may never get removed.
> 
>
> Key: KAFKA-3137
> URL: https://issues.apache.org/jira/browse/KAFKA-3137
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>
> I spoke about this with [~junrao]. I haven't tried to reproduce this, but Jun 
> said that it looks like this is possible, so I'm filing it.
> Delete tombstones in log compacted topics are deleted after 
> delete.retention.ms (at the topic level) or log.cleaner.delete.retention.ms 
> (at the broker level).
> However, we don't have per-message timestamps (at least until KIP-32 is 
> implemented). So the timestamp of the log segment file is used as a proxy. 
> However, the modification time of the log segment changes whenever a 
> compaction run happens.
> It's possible then that if log compaction happens very frequently that 
> delete.retention.ms will never be reached. In that case, the delete 
> tombstones would stay around longer than the user expected. 
> I believe that means that log compaction would have to happen more frequently 
> than delete.retention.ms. The frequency of log compaction is some calculation 
> based on segment size, the criteria for segment roll (time or bytes), the 
> min.cleanable.dirty.ratio, as well as the amount of traffic coming into the 
> log compacted topic. So it's possible, but I'm not sure how likely.
> And I would imagine that this can't be fixed until KIP-32 is available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-62: Allow consumer to send heartbeats from a background thread

[Jason Gustafson]

Hey Grant,

Thanks for the feedback. I'm definitely open to including heartbeat() in
this KIP. One thing we should be clear about is what the behavior of
heartbeat() should be when the group begins rebalancing. I think there are
basically two options:

1. heartbeat() simply keeps heartbeating even if the group has started
rebalancing.
2. heartbeat() completes the rebalance itself.

With the first option, when processing takes longer than the rebalance
timeout, the member will fall out of the group which will cause an offset
commit failure when it finally finishes. However, if processing finishes
before the rebalance completes, then offsets can still be committed. On the
other hand, if heartbeat() completes the rebalance itself, then you'll
definitely see the offset commit failure for any records being processed.
So the first option is sort of biased toward processing completion while
the latter is biased toward rebalance completion.

I'm definitely not a fan of second option since it takes away the choice to
finish processing before rejoining. However, I do see some benefit in the
first option if the user wants to keep rebalance time low and doesn't mind
being kicked out of the group if processing takes longer during a
rebalance. This may be a reasonable tradeoff since consumer groups are
presumed to be stable most of the time. A better option in that case might
be to expose the rebalance timeout to the user directly since it would
allow the user to use an essentially unbounded process.timeout.ms for
highly variant processing while still keeping rebalance time limited. Of
course, it would be another timeout for the user to understand...

Thanks,
Jason

On Thu, May 26, 2016 at 8:19 AM, Grant Henke  wrote:

> Hi Jason,
>
> Thanks for writing up a proposal (and a thorough one)! This is something
> that I had been thinking about this week too as I have run into it more
> than a handful of times now.
>
> I like the idea of having a larger processing timeout, that timeout in
> unison with max.poll.records should in many cases provide a reasonable
> assurance that the consumer will stay alive.
>
> In rejected alternatives "Add a separate API the user can call to indicate
> liveness" is listed. I think a heartbeat api could be added along with
> these new timeout configurations and used for "advanced" use cases where
> the processing time could be highly variant and less predictable. I think a
> place where we might use the heartbeat api in Kafka is MirrorMaker.
>
> Today, I have seen people trying to find ways to leverage the existing api
> to "force" heartbeats by:
>
> 1. Calling poll to get the batch of records to process
> 2. Call pause on all partitions
> 3. Process the record batch
> 3a. While processing periodically call poll (which is essentially just
> heartbeat since it returns no records and is paused)
> 4. Commit offsets and un-pause
> 5. Repeat from 1
>
> Thanks,
> Grant
>
>
>
>
>
>
> On Wed, May 25, 2016 at 6:32 PM, Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > One of the persistent problems we see with the new consumer is the use of
> > the session timeout in order to ensure progress. Whenever there is a
> delay
> > in message processing which exceeds the session timeout, no heartbeats
> can
> > be sent and the consumer is removed from the group. We seem to hit this
> > problem everywhere the consumer is used (including Kafka Connect and
> Kafka
> > Streams) and we don't always have a great solution. I've written a KIP to
> > address this problem here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> > .
> > Have a look and let me know what you think.
> >
> > Thanks,
> > Jason
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

[jira] [Created] (KAFKA-3759) Incorrect JDBC credentials cause Connect worker to permanently fail

[Lars George (JIRA)]

Lars George created KAFKA-3759:
--

 Summary: Incorrect JDBC credentials cause Connect worker to 
permanently fail
 Key: KAFKA-3759
 URL: https://issues.apache.org/jira/browse/KAFKA-3759
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 0.9.0.1
 Environment: I work on a cloudera quickstart VM. I installed kafka / 
confluent via yum :
 
[cloudera@quickstart confluent]$ yum list installed | grep confluent
confluent-camus.noarch  2.0.1-1   @confluent-2.0   
confluent-common.noarch 2.0.1-1   @confluent-2.0   
confluent-kafka-2.11.7.noarch   0.9.0.1-1 @confluent-2.0   
confluent-kafka-connect-hdfs.noarch 2.0.1-1   @confluent-2.0   
confluent-kafka-connect-jdbc.noarch 2.0.1-1   @confluent-2.0   
confluent-kafka-rest.noarch 2.0.1-1   @confluent-2.0   
confluent-rest-utils.noarch 2.0.1-1   @confluent-2.0   
confluent-schema-registry.noarch2.0.1-1   @confluent-2.0  
 
Reporter: Lars George
Assignee: Ewen Cheslack-Postava


As reported by our peers:

All the following steps are executed in one cloudera instance, each in a 
different terminal.
- I start the kafka server with the (unchanged) properties file in 
/etc/kafka/server.properties
- I start the schema-registry with the (unchanged) properties file in 
/etc/schema-registry/schema-registry.properties
- I start the connect worker with the properties file 
/etc/schema-registry/connect-avro-distributed.properties
HOWEVER I have changed the following 2 properties for my test:
config.storage.topic=connect-configs-broken
offset.storage.topic=connect-offsets-broken
 
Now I execute a shell script that uses the REST API to post a connector 
configuration to the worker. Here the snippet from the script
that shows the connector config:

{noformat} 
==
read -resp $'please enter the password for the postgresql user \n' 
postgres_password
 
echo posting connector config into the kafka cluster
data="{
\"name\": \"projectx-postgres-test\",
\"config\": {
   
\"connection.url\":\"jdbc:postgresql://$postgres_db_url/$postgres_db_name?user=$postgres_user=$postgres_password\",
   
\"connector.class\":\"io.confluent.connect.jdbc.JdbcSourceConnector\",
   \"tasks.max\":\"1\",
   \"mode\":\"incrementing\",
   \"incrementing.column.name\":\"id\",
   \"topic.prefix\":\"test-postgres-jdbc-\"
}
}"
url=http://$kafka_connect_rest_url/connectors
ctype="Content-Type: application/json"
curl -X POST $url -H "$ctype" --data "$data"
 
==
{noformat}
 
- I exectute the script with correct connection settings, but incorrect 
postgresql username / password.
- I receive the following answer from the REST API:

{noformat}
{"name":"projectx-postgres-test","config":{"connection.url":"jdbc:postgresql://localhost/connect_test?user=cloudera=wrongPW","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","tasks.max":"1","mode":"incrementing","incrementing.column.name":"id","topic.prefix":"test-postgres-jdbc-","name":"projectx-postgres-test"},"tasks":[]}
{noformat}
 
Now the connect worker stops and I see the following error and stack trace:
 
{noformat}
[2016-05-23 05:53:43,966] INFO 127.0.0.1 - - [23/May/2016:12:53:41 +] "POST 
/connectors HTTP/1.1" 201 343  2500 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-05-23 05:53:44,656] ERROR Couldn't open connection to 
jdbc:postgresql://localhost/connect_test?user=cloudera=wrongPW: 
org.postgresql.util.PSQLException: FATAL: password authentication failed for 
user "cloudera" (io.confluent.connect.jdbc.JdbcSourceConnector:76)
[2016-05-23 05:53:44,663] ERROR Uncaught exception in herder work thread, 
exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:166)
org.apache.kafka.connect.errors.ConnectException: Connector threw an exception 
while starting
at org.apache.kafka.connect.runtime.Worker.addConnector(Worker.java:188)
at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:670)
at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startWork(DistributedHerder.java:642)
at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.handleRebalanceCompleted(DistributedHerder.java:600)
at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:184)
at 

Build failed in Jenkins: kafka-trunk-jdk8 #652

[Apache Jenkins Server]

See 

Changes:

[cshapi] MINOR: Fix wrong comments

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-2 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision d5366471d2c1a611f2241362041debc0118e5422 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d5366471d2c1a611f2241362041debc0118e5422
 > git rev-list c699b1a914260b7d6fd84596f78128369d2bf2bf # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson6105762081899470144.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 12.844 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson6209225260130203917.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJavawarning: [options] bootstrap class path 
not set in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 

[GitHub] kafka pull request: MINOR: Fix tracing in KafkaApis.handle()

[mimaison]

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/1435

MINOR: Fix tracing in KafkaApis.handle()

requestObj() returns null for the o.a.k.c.requests objects so use header() 
for these.

Once all the requests will have been replaced by o.a.k.c.requests objects, 
we should be able to clean that up, but in the meantime it's useful to trace 
both.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka kafkaapis_trace

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1435.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1435


commit 1fb627c8a5f57b4c8762f351564623034daf2158
Author: Mickael Maison 
Date:   2016-05-26T17:36:55Z

MINOR: Fix tracing in KafkaApis.handle()

requestObj returns null for the new Java request objects so use header for 
these.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1315

[Apache Jenkins Server]

See 

Changes:

[junrao] Setting broker state as running after publishing to ZK

[cshapi] MINOR: Use `--force` instead of `--yes` in `AclCommand`

[cshapi] MINOR: Fix wrong comments

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision d5366471d2c1a611f2241362041debc0118e5422 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d5366471d2c1a611f2241362041debc0118e5422
 > git rev-list 62dc1afb69369c64207991ba59bcd203505d37ea # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson7517580922073119382.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 22.05 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson4203509840360405743.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk7:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 22.556 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302542#comment-15302542
 ] 

Greg Fodor commented on KAFKA-3758:
---

Also, the log is truncated at the top to the point where we shut the broker 
off. If there's additional useful information in the log before that you think 
we could share, happy to attach.

> KStream job fails to recover after Kafka broker stopped
> ---
>
> Key: KAFKA-3758
> URL: https://issues.apache.org/jira/browse/KAFKA-3758
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
> Attachments: muon.log.1.gz
>
>
> We've been doing some testing of a fairly complex KStreams job and under load 
> it seems the job fails to rebalance + recover if we shut down one of the 
> kafka brokers. The test we were running had a 3-node kafka cluster where each 
> topic had at least a replication factor of 2, and we terminated one of the 
> nodes.
> Attached is the full log, the root exception seems to be contention on the 
> lock on the state directory. The job continues to try to recover but throws 
> errors relating to locks over and over. Restarting the job itself resolves 
> the problem.
>  1702 org.apache.kafka.streams.errors.ProcessorStateException: Error while 
> creating the state manager
>  1703 at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
>  1704 at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
>  1705 at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
>  1706 at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
>  1707 at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
>  1708 at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
>  1709 at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
>  1710 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
>  1711 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
>  1712 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1713 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1714 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
>  1715 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1716 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1717 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
>  1718 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
>  1719 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
>  1720 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
>  1721 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
>  1722 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1723 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1724 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
>  1725 at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>  1726 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>  1727 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>  1728 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>  1729 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>  1730 at 
> 

[jira] [Updated] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

[Greg Fodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Fodor updated KAFKA-3758:
--
Attachment: muon.log.1.gz

> KStream job fails to recover after Kafka broker stopped
> ---
>
> Key: KAFKA-3758
> URL: https://issues.apache.org/jira/browse/KAFKA-3758
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
> Attachments: muon.log.1.gz
>
>
> We've been doing some testing of a fairly complex KStreams job and under load 
> it seems the job fails to rebalance + recover if we shut down one of the 
> kafka brokers. The test we were running had a 3-node kafka cluster where each 
> topic had at least a replication factor of 2, and we terminated one of the 
> nodes.
> Attached is the full log, the root exception seems to be contention on the 
> lock on the state directory. The job continues to try to recover but throws 
> errors relating to locks over and over. Restarting the job itself resolves 
> the problem.
>  1702 org.apache.kafka.streams.errors.ProcessorStateException: Error while 
> creating the state manager
>  1703 at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
>  1704 at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
>  1705 at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
>  1706 at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
>  1707 at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
>  1708 at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
>  1709 at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
>  1710 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
>  1711 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
>  1712 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1713 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1714 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
>  1715 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1716 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1717 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
>  1718 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
>  1719 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
>  1720 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
>  1721 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
>  1722 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1723 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1724 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
>  1725 at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>  1726 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>  1727 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>  1728 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>  1729 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>  1730 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
>  1731 at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
>  1732 at 

[jira] [Created] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

[Greg Fodor (JIRA)]

Greg Fodor created KAFKA-3758:
-

 Summary: KStream job fails to recover after Kafka broker stopped
 Key: KAFKA-3758
 URL: https://issues.apache.org/jira/browse/KAFKA-3758
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.0.0
Reporter: Greg Fodor
Assignee: Guozhang Wang


We've been doing some testing of a fairly complex KStreams job and under load 
it seems the job fails to rebalance + recover if we shut down one of the kafka 
brokers. The test we were running had a 3-node kafka cluster where each topic 
had at least a replication factor of 2, and we terminated one of the nodes.

Attached is the full log, the root exception seems to be contention on the lock 
on the state directory. The job continues to try to recover but throws errors 
relating to locks over and over. Restarting the job itself resolves the problem.

 1702 org.apache.kafka.streams.errors.ProcessorStateException: Error while 
creating the state manager
 1703 at 
org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
 1704 at 
org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
 1705 at 
org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
 1706 at 
org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
 1707 at 
org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
 1708 at 
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
 1709 at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
 1710 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
 1711 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
 1712 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
 1713 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
 1714 at 
org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
 1715 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
 1716 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
 1717 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
 1718 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
 1719 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
 1720 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
 1721 at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
 1722 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
 1723 at 
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
 1724 at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
 1725 at 
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
 1726 at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
 1727 at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
 1728 at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
 1729 at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
 1730 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
 1731 at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
 1732 at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977)
 1733 at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
 1734 at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295)
 1735 at 

[jira] [Created] (KAFKA-3757) Add new client to "clients" page please

[Vadim Chekan (JIRA)]

Vadim Chekan created KAFKA-3757:
---

 Summary: Add new client to "clients" page please
 Key: KAFKA-3757
 URL: https://issues.apache.org/jira/browse/KAFKA-3757
 Project: Kafka
  Issue Type: Task
  Components: website
Affects Versions: 0.10.0.0
Reporter: Vadim Chekan
Priority: Minor


I'd like kafka4net client to be added to "clients" page:
https://cwiki.apache.org/confluence/display/KAFKA/Clients

This is C# client, asynchronous, all 3 compressions supported (read and write), 
tracks leader partition changes transparently, long time in production.
Maintainer: https://github.com/vchekan/
License: Apache-2.0
Repository: https://github.com/ntent-ad/kafka4net

I've registered on the cwiki.apache.org site, but I do not have permissions to 
edit the page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Fix wrong comments

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1198


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #651

[Apache Jenkins Server]

See 

Changes:

[junrao] Setting broker state as running after publishing to ZK

[cshapi] MINOR: Use `--force` instead of `--yes` in `AclCommand`

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-us1 (Ubuntu ubuntu ubuntu-us golang-ppa) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision c699b1a914260b7d6fd84596f78128369d2bf2bf 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c699b1a914260b7d6fd84596f78128369d2bf2bf
 > git rev-list 62dc1afb69369c64207991ba59bcd203505d37ea # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7410673762475568522.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 19.11 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson3284727289890076004.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 14.552 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK1_8_0_66_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66

[jira] [Commented] (KAFKA-3689) ERROR Processor got uncaught exception. (kafka.network.Processor)

[Buvaneswari Ramanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302469#comment-15302469
 ] 

Buvaneswari Ramanan commented on KAFKA-3689:


The error manifests in 0.10.0.0 RC6 as well

Here is the scenario:

Identical 4 broker cluster, 8000 topics with single partition & replication 
factor of 3, 200 bytes/sec into each topic.
One after other, 3 brokers were restarted and while they were coming to steady 
state, the 4th broker logged a million of this error message in its log. Until 
we restarted the 4th broker, this error message repeated. 

Here is the exact message this time:

[2016-05-24 09:17:31,328] ERROR Processor got uncaught exception. 
(kafka.network.Processor)
java.lang.IllegalArgumentException: Attempted to decrease connection count for 
address with no connections, address: /X.Y.Z.140 (actual network address masked)
   at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:594)
   at 
kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:594)
   at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
   at scala.collection.AbstractMap.getOrElse(Map.scala:58)
   at kafka.network.ConnectionQuotas.dec(SocketServer.scala:593)
   at 
kafka.network.Processor$$anonfun$processDisconnected$1.apply(SocketServer.scala:517)
   at 
kafka.network.Processor$$anonfun$processDisconnected$1.apply(SocketServer.scala:511)
   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
   at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
   at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
   at 
kafka.network.Processor.processDisconnected(SocketServer.scala:511)
   at kafka.network.Processor.run(SocketServer.scala:415)
   at java.lang.Thread.run(Thread.java:745)


> ERROR Processor got uncaught exception. (kafka.network.Processor)
> -
>
> Key: KAFKA-3689
> URL: https://issues.apache.org/jira/browse/KAFKA-3689
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
> Environment: ubuntu 14.04,
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.2)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> 3 broker cluster (all 3 servers identical -  Intel Xeon E5-2670 @2.6GHz, 
> 8cores, 16 threads 64 GB RAM & 1 TB Disk)
> Kafka Cluster is managed by 3 server ZK cluster (these servers are different 
> from Kafka broker servers). All 6 servers are connected via 10G switch. 
> Producers run from external servers.
>Reporter: Buvaneswari Ramanan
>Assignee: Jun Rao
>Priority: Minor
> Fix For: 0.10.1.0, 0.10.0.1
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> As per Ismael Juma's suggestion in email thread to us...@kafka.apache.org 
> with the same subject, I am creating this bug report.
> The following error occurs in one of the brokers in our 3 broker cluster, 
> which serves about 8000 topics. These topics are single partitioned with a 
> replication factor = 3. Each topic gets data at a low rate  – 200 bytes per 
> sec.  Leaders are balanced across the topics.
> Producers run from external servers (4 Ubuntu servers with same config as the 
> brokers), each producing to 2000 topics utilizing kafka-python library.
> This error message occurs repeatedly in one of the servers. Between the hours 
> of 10:30am and 1:30pm on 5/9/16, there were about 10 Million such 
> occurrences. This was right after a cluster restart.
> This is not the first time we got this error in this broker. In those 
> instances, error occurred hours / days after cluster restart.
> =
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count 
> for address with no connections, address: /X.Y.Z.144 (actual network address 
> masked)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at 

[GitHub] kafka pull request: MINOR: Use `--force` instead of `--yes` in `Ac...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1430


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Setting broker state as running after publishi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1426


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka Wiki

[Gwen Shapira]

Done :)

On Thu, May 26, 2016 at 7:48 AM, Matthias J. Sax 
wrote:

> Hi,
>
> can I get write access to the Kafka Wiki? Username is: mjsax
>
> Thanks.
>
> -Matthias
>
>

Re: [DISCUSS] KIP-62: Allow consumer to send heartbeats from a background thread

[Grant Henke]

Hi Jason,

Thanks for writing up a proposal (and a thorough one)! This is something
that I had been thinking about this week too as I have run into it more
than a handful of times now.

I like the idea of having a larger processing timeout, that timeout in
unison with max.poll.records should in many cases provide a reasonable
assurance that the consumer will stay alive.

In rejected alternatives "Add a separate API the user can call to indicate
liveness" is listed. I think a heartbeat api could be added along with
these new timeout configurations and used for "advanced" use cases where
the processing time could be highly variant and less predictable. I think a
place where we might use the heartbeat api in Kafka is MirrorMaker.

Today, I have seen people trying to find ways to leverage the existing api
to "force" heartbeats by:

1. Calling poll to get the batch of records to process
2. Call pause on all partitions
3. Process the record batch
3a. While processing periodically call poll (which is essentially just
heartbeat since it returns no records and is paused)
4. Commit offsets and un-pause
5. Repeat from 1

Thanks,
Grant






On Wed, May 25, 2016 at 6:32 PM, Jason Gustafson  wrote:

> Hi All,
>
> One of the persistent problems we see with the new consumer is the use of
> the session timeout in order to ensure progress. Whenever there is a delay
> in message processing which exceeds the session timeout, no heartbeats can
> be sent and the consumer is removed from the group. We seem to hit this
> problem everywhere the consumer is used (including Kafka Connect and Kafka
> Streams) and we don't always have a great solution. I've written a KIP to
> address this problem here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> .
> Have a look and let me know what you think.
>
> Thanks,
> Jason
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Kafka Wiki

[Matthias J. Sax]

Hi,

can I get write access to the Kafka Wiki? Username is: mjsax

Thanks.

-Matthias



signature.asc
Description: OpenPGP digital signature

Re: [kafka-clients] Adding new client to wiki

[Dana Powers]

I believe wiki requests usually go to the kafka dev mailing list (cc'd)
On May 26, 2016 6:51 AM, "Vijay Jadhav"  wrote:

>
>
> Hi,
>
> Can someone point out what is procedure for adding "libasynckafkaclient
> " (C++ based single
> threaded asynchronous library ) : to client wiki
>  page ?
>
> Thanks
> Vijay
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/c47b01c1-4b6f-42b1-887f-172e9f5c4403%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[Dustin Cote (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302150#comment-15302150
 ] 

Dustin Cote commented on KAFKA-2394:


[~hachikuji] the attached PR removes references to the DailyRollingFileAppender 
because it's not just at risk when you have spamming logging, but over the long 
term, the DailyRollingFileAppender has no cleanup policy that I know of by 
default.  It also has known issues with [losing log 
messages|https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html].
  I think it's good to scrub DailyRollingFileAppender completely and notify of 
the format change for logging or add the log4j extras to the dependency list to 
match the formatting.  I prefer just changing the naming convention of the 
files but am certainly open to discussion. Thanks!

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: jin xing
>Priority: Minor
>  Labels: newbie
> Attachments: log4j.properties.patch
>
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302104#comment-15302104
 ] 

ASF GitHub Bot commented on KAFKA-2394:
---

GitHub user cotedm opened a pull request:

https://github.com/apache/kafka/pull/1434

KAFKA-2394: move to RollingFileAppender by default for log4j

This PR sets up the default max log size to be about 1GB.  That default 
should probably be vetted.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cotedm/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1434


commit 6eab7890db19cf0d873530278b8cd43877063f69
Author: Dustin Cote 
Date:   2016-05-26T13:52:35Z

move to RollingFileAppender by default for log4j




> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: jin xing
>Priority: Minor
>  Labels: newbie
> Attachments: log4j.properties.patch
>
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2394: move to RollingFileAppender by def...

[cotedm]

GitHub user cotedm opened a pull request:

https://github.com/apache/kafka/pull/1434

KAFKA-2394: move to RollingFileAppender by default for log4j

This PR sets up the default max log size to be about 1GB.  That default 
should probably be vetted.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cotedm/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1434


commit 6eab7890db19cf0d873530278b8cd43877063f69
Author: Dustin Cote 
Date:   2016-05-26T13:52:35Z

move to RollingFileAppender by default for log4j




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[Dustin Cote (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302098#comment-15302098
 ] 

Dustin Cote commented on KAFKA-2394:


[~jinxing6...@126.com] I'll go ahead and take this one to move it forward.  
Please let me know if you end up having time to work on it.  Thanks!

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: jin xing
>Priority: Minor
>  Labels: newbie
> Attachments: log4j.properties.patch
>
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

KAFKA-3744: Message format to identify serializer

[David Kay]

All, I plan to submit a KIP to begin discussion on 
https://issues.apache.org/jira/browse/KAFKA-3744 and the associated Pull 
Request https://github.com/apache/kafka/pull/1419.  I'll notify this list when 
the KIP is submitted.

Please discard my previous message containing bogus subject and links.

KAFKA-3698: Message format to identify serializer

[David Kay]

All, I plan to submit a KIP to begin discussion on 
https://issues.apache.org/jira/browse/KAFKA-3698 and the associated PR
https://github.com/apache/kafka/commit/27a19b964af35390d78e1b3b50bc03d23327f4d0.
  I'll notify this list when the KIP is submitted.
[https://avatars2.githubusercontent.com/u/5778611?v=3=200]

Build failed in Jenkins: kafka-trunk-jdk8 #650

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3718; propagate all KafkaConfig __consumer_offsets configs to

--
[...truncated 737 lines...]
kafka.api.SaslSslConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.SaslSslConsumerTest > testUnsubscribeTopic PASSED

kafka.api.SaslSslConsumerTest > testListTopics PASSED

kafka.api.SaslSslConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslSslConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.SaslSslConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoConsumeAcl PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testProduceConsume PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoProduceAcl PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.message.MessageWriterTest > testWithNoCompressionAttribute PASSED

kafka.message.MessageWriterTest > testWithCompressionAttribute PASSED

kafka.message.MessageWriterTest > testBufferingOutputStream PASSED

kafka.message.MessageWriterTest > testWithKey PASSED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.message.MessageTest > testMessageFormatConversion PASSED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testLZ4FramingV0 PASSED

kafka.message.MessageCompressionTest > testLZ4FramingV1 PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testOffsetAssignmentAfterMessageFormatConversion PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testAbsoluteOffsetAssignment PASSED

kafka.message.ByteBufferMessageSetTest > testCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testInvalidCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testLogAppendTime PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testRelativeOffsetAssignment PASSED

kafka.producer.ProducerTest > testSendToNewTopic PASSED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout PASSED

kafka.producer.ProducerTest > testSendNullMessage PASSED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.producer.SyncProducerTest > testNotEnoughReplicas PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.producer.SyncProducerTest > testProduceCorrectlyReceivesResponse PASSED

kafka.controller.ControllerFailoverTest > 

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Edoardo Comar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301940#comment-15301940
 ] 

Edoardo Comar commented on KAFKA-3177:
--

see also Consumer.poll() stuck in loop on non-existent topic manually assigned 
: https://issues.apache.org/jira/browse/KAFKA-3727

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.1.0
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1379) Partition reassignment resets clock for time-based retention

[Moritz Siuts (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301936#comment-15301936
 ] 

Moritz Siuts commented on KAFKA-1379:
-

>From the user-mailinglist:

{quote}
We’ve recently upgraded to 0.9.  In 0.8, when we restarted a broker, data
log file mtimes were not changed.  In 0.9, any data log file that was on
disk before the broker has it’s mtime modified to the time of the broker
restart.
{quote}

A workaround can be to set {{retention.bytes}} on a topic level, like this:

{noformat}
./bin/kafka-topics.sh --zookeeper X.X.X.X:2181/kafka -alter --config 
retention.bytes=500 –topic my_topic
{noformat}

The settings controls the max size in bytes of a partition oft he specified 
topic. So you can find a good size by checking the size of a partition with 
{{du -b}} and use this value.

> Partition reassignment resets clock for time-based retention
> 
>
> Key: KAFKA-1379
> URL: https://issues.apache.org/jira/browse/KAFKA-1379
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>
> Since retention is driven off mod-times reassigned partitions will result in
> data that has been on a leader to be retained for another full retention
> cycle. E.g., if retention is seven days and you reassign partitions on the
> sixth day then those partitions will remain on the replicas for another
> seven days.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.0-jdk7 #109

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3718; propagate all KafkaConfig __consumer_offsets configs to

--
Started by an SCM change
Started by an SCM change
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H10 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/0.10.0^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/0.10.0^{commit} # timeout=10
Checking out Revision 5f9446498e24afdb1149af53583c0ab23345d965 
(refs/remotes/origin/0.10.0)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5f9446498e24afdb1149af53583c0ab23345d965
 > git rev-list cb8b6c07140b9d0cca2f17c2fdcdd0f1c7248b99 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-0.10.0-jdk7] $ /bin/bash -xe /tmp/hudson1142039052049314517.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 34.123 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-0.10.0-jdk7] $ /bin/bash -xe /tmp/hudson1116843346304128315.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-0.10.0-jdk7:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
org.gradle.api.internal.changedetection.state.FileCollectionSnapshotImpl cannot 
be cast to 
org.gradle.api.internal.changedetection.state.OutputFilesCollectionSnapshotter$OutputFilesSnapshot

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 28.666 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51

Build failed in Jenkins: kafka-trunk-jdk7 #1314

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3718; propagate all KafkaConfig __consumer_offsets configs to

--
Started by an SCM change
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 62dc1afb69369c64207991ba59bcd203505d37ea 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 62dc1afb69369c64207991ba59bcd203505d37ea
 > git rev-list 5653249e08b1050f0e2081bf1dcc50324b3ec409 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson5164189956570980803.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 18.759 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson8361133565869479220.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
Build file ': 
line 231
useAnt has been deprecated and is scheduled to be removed in Gradle 3.0. The 
Ant-Based Scala compiler is deprecated, please see 
https://docs.gradle.org/current/userguide/scala_plugin.html.
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk7:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 20.257 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51

[GitHub] kafka pull request: delete unused code

[leisore]

GitHub user leisore opened a pull request:

https://github.com/apache/kafka/pull/1433

delete unused code



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/leisore/kafka master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1433


commit c8c5933088496cb01d3fd6416665a4826b6a3dfa
Author: leisore 
Date:   2016-05-26T10:20:08Z

delete unused code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3718) propagate all KafkaConfig __consumer_offsets configs to OffsetConfig instantiation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301740#comment-15301740
 ] 

ASF GitHub Bot commented on KAFKA-3718:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1394


> propagate all KafkaConfig __consumer_offsets configs to OffsetConfig 
> instantiation
> --
>
> Key: KAFKA-3718
> URL: https://issues.apache.org/jira/browse/KAFKA-3718
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Onur Karaman
> Fix For: 0.10.0.1
>
>
> Kafka has two configurable compression codecs: the one used by the client 
> (source codec) and the one finally used when storing into the log (target 
> codec). The target codec defaults to KafkaConfig.compressionType and can be 
> dynamically configured through zookeeper.
> The GroupCoordinator appends group membership information into the 
> __consumer_offsets topic by:
> 1. making a message with group membership information
> 2. making a MessageSet with the single message compressed with the source 
> codec
> 3. doing a log.append on the MessageSet
> Without this patch, KafkaConfig.offsetsTopicCompressionCodec doesn't get 
> propagated to OffsetConfig instantiation, so GroupMetadataManager uses a 
> source codec of NoCompressionCodec when making the MessageSet. Let's say we 
> have enough group information such that the message formed exceeds 
> KafkaConfig.messageMaxBytes before compression but would fall below the 
> threshold after compression using our source codec. Even if we had 
> dynamically configured __consumer_offsets with our favorite compression 
> codec, the log.append will throw RecordTooLargeException during 
> analyzeAndValidateMessageSet since the message was unexpectedly uncompressed 
> instead of having been compressed with the source codec defined by 
> KafkaConfig.offsetsTopicCompressionCodec.
> NOTE: even after this issue is resolved, preliminary tests show that LinkedIn 
> will still hit RecordTooLargeException with large groups that consume many 
> topics (like MirrorMakers with wildcard consumption of .*) since fully 
> expanded subscription and assignment state for each member is put into a 
> single record. But this is a first step in the right direction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka broker crash - broker id then changed

[Ben Davison]

Possibly tmp got cleaned up?

Seems like one of the log files where deleted while a producer was writing
messages to it:

On Thursday, 26 May 2016, cs user  wrote:

> Hi All,
>
> We are running Kafka version 0.9.0.1, at the time the brokers crashed
> yesterday we were running in a 2 mode cluster. This has now been increased
> to 3.
>
> We are not specifying a broker id and relying on kafka generating one.
>
> After the brokers crashed (at exactly the same time) we left kafka stopped
> for a while. After kafka was started back up, the broker id's on both
> servers were incremented, they were 1001/1002 and they flipped to
> 1003/1004. This seemed to cause some problems as partitions were assigned
> to broker id's which it believed had disappeared and so were not
> recoverable.
>
> We noticed that the broker id's are actually stored in:
>
> /tmp/kafka-logs/meta.properties
>
> So we set these back to what they were and restarted. Is there a reason why
> these would change?
>
> Below are the error logs from each server:
>
> Server 1
>
> [2016-05-25 09:05:52,827] INFO [ReplicaFetcherManager on broker 1002]
> Removed fetcher for partitions [Topic1Heartbeat,1]
> (kafka.server.ReplicaFetcherManager)
> [2016-05-25 09:05:52,831] INFO Completed load of log Topic1Heartbeat-1 with
> log end offset 0 (kafka.log.Log)
> [2016-05-25 09:05:52,831] INFO Created log for partition
> [Topic1Heartbeat,1] in /tmp/kafka-logs with properties {compression.type ->
> producer, file.delete.delay.ms -> 6, max.message.bytes -> 112,
> min.insync.replicas -> 1, segment.
> jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5,
> index.interval.bytes -> 4096, unclean.leader.election.enable -> true,
> retention.bytes -> -1, delete.retention.ms -> 8640, cleanup.policy ->
> delete, flush.ms -> 9
> 223372036854775807, segment.ms -> 60480, segment.bytes -> 1073741824,
> retention.ms -> 60480, segment.index.bytes -> 10485760, flush.messages
> -> 9223372036854775807}. (kafka.log.LogManager)
> [2016-05-25 09:05:52,831] INFO Partition [Topic1Heartbeat,1] on broker
> 1002: No checkpointed highwatermark is found for partition
> [Topic1Heartbeat,1] (kafka.cluster.Partition)
> [2016-05-25 09:14:12,189] INFO [GroupCoordinator 1002]: Preparing to
> restabilize group Topic1 with old generation 0
> (kafka.coordinator.GroupCoordinator)
> [2016-05-25 09:14:12,190] INFO [GroupCoordinator 1002]: Stabilized group
> Topic1 generation 1 (kafka.coordinator.GroupCoordinator)
> [2016-05-25 09:14:12,195] INFO [GroupCoordinator 1002]: Assignment received
> from leader for group Topic1 for generation 1
> (kafka.coordinator.GroupCoordinator)
> [2016-05-25 09:14:12,749] FATAL [Replica Manager on Broker 1002]: Halting
> due to unrecoverable I/O error while handling produce request:
>  (kafka.server.ReplicaManager)
> kafka.common.KafkaStorageException: I/O exception in append to log
> '__consumer_offsets-0'
> at kafka.log.Log.append(Log.scala:318)
> at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:442)
> at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:428)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:268)
> at
> kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:428)
> at
>
> kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:401)
> at
>
> kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:386)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:386)
> at
> kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:322)
> at
>
> kafka.coordinator.GroupMetadataManager.store(GroupMetadataManager.scala:228)
> at
>
> kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429)
> at
>
> kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429)
> at scala.Option.foreach(Option.scala:257)
> at
>
> kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:429)
> at
> kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:280)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:76)
> at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> 

[jira] [Resolved] (KAFKA-3718) propagate all KafkaConfig __consumer_offsets configs to OffsetConfig instantiation

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3718.

Resolution: Fixed
  Reviewer: Ismael Juma

> propagate all KafkaConfig __consumer_offsets configs to OffsetConfig 
> instantiation
> --
>
> Key: KAFKA-3718
> URL: https://issues.apache.org/jira/browse/KAFKA-3718
> Project: Kafka
>  Issue Type: Bug
>Reporter: Onur Karaman
>Assignee: Onur Karaman
> Fix For: 0.10.0.1
>
>
> Kafka has two configurable compression codecs: the one used by the client 
> (source codec) and the one finally used when storing into the log (target 
> codec). The target codec defaults to KafkaConfig.compressionType and can be 
> dynamically configured through zookeeper.
> The GroupCoordinator appends group membership information into the 
> __consumer_offsets topic by:
> 1. making a message with group membership information
> 2. making a MessageSet with the single message compressed with the source 
> codec
> 3. doing a log.append on the MessageSet
> Without this patch, KafkaConfig.offsetsTopicCompressionCodec doesn't get 
> propagated to OffsetConfig instantiation, so GroupMetadataManager uses a 
> source codec of NoCompressionCodec when making the MessageSet. Let's say we 
> have enough group information such that the message formed exceeds 
> KafkaConfig.messageMaxBytes before compression but would fall below the 
> threshold after compression using our source codec. Even if we had 
> dynamically configured __consumer_offsets with our favorite compression 
> codec, the log.append will throw RecordTooLargeException during 
> analyzeAndValidateMessageSet since the message was unexpectedly uncompressed 
> instead of having been compressed with the source codec defined by 
> KafkaConfig.offsetsTopicCompressionCodec.
> NOTE: even after this issue is resolved, preliminary tests show that LinkedIn 
> will still hit RecordTooLargeException with large groups that consume many 
> topics (like MirrorMakers with wildcard consumption of .*) since fully 
> expanded subscription and assignment state for each member is put into a 
> single record. But this is a first step in the right direction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3718: propagate all KafkaConfig __consum...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1394


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301718#comment-15301718
 ] 

Eno Thereska commented on KAFKA-3511:
-

>From feedback, it's probably best to hold off from this one for now. Users can 
>pass in their own aggregators currently (and in Java 8 they will be able to 
>also use lambda functions). It is not 100% clear what the value of providing 
>built-in aggregators would be when it is really easy for users to write their 
>own (e.g., count or sum aggregator).

> Add common aggregation functions like Sum and Avg as build-ins in Kafka 
> Streams DSL
> ---
>
> Key: KAFKA-3511
> URL: https://issues.apache.org/jira/browse/KAFKA-3511
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: api
> Fix For: 0.10.1.0
>
>
> Currently we have the following aggregation APIs in the Streams DSL:
> {code}
> KStream.aggregateByKey(..)
> KStream.reduceByKey(..)
> KStream.countByKey(..)
> KTable.groupBy(...).aggregate(..)
> KTable.groupBy(...).reduce(..)
> KTable.groupBy(...).count(..)
> {code}
> And it is better to add common aggregation functions like Sum and Avg as 
> built-in into the Streams DSL. A few questions to ask though:
> 1. Should we add those built-in functions as, for example 
> {{KTable.groupBy(...).sum(...)} or {{KTable.groupBy(...).aggregate(SUM, 
> ...)}}. Please see the comments below for detailed pros and cons.
> 2. If we go with the second option above, should we replace the countByKey / 
> count operators with aggregate(COUNT) as well? Personally I (Guozhang) feel 
> it is not necessary, as COUNT is a special aggregate function since we do not 
> need to map on any value fields; this is the same approach as in Spark as 
> well, where Count is built-in as first-citizen in the DSL, and others are 
> built-in as {{aggregate(SUM)}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work stopped] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3511 stopped by Eno Thereska.
---
> Add common aggregation functions like Sum and Avg as build-ins in Kafka 
> Streams DSL
> ---
>
> Key: KAFKA-3511
> URL: https://issues.apache.org/jira/browse/KAFKA-3511
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: api
> Fix For: 0.10.1.0
>
>
> Currently we have the following aggregation APIs in the Streams DSL:
> {code}
> KStream.aggregateByKey(..)
> KStream.reduceByKey(..)
> KStream.countByKey(..)
> KTable.groupBy(...).aggregate(..)
> KTable.groupBy(...).reduce(..)
> KTable.groupBy(...).count(..)
> {code}
> And it is better to add common aggregation functions like Sum and Avg as 
> built-in into the Streams DSL. A few questions to ask though:
> 1. Should we add those built-in functions as, for example 
> {{KTable.groupBy(...).sum(...)} or {{KTable.groupBy(...).aggregate(SUM, 
> ...)}}. Please see the comments below for detailed pros and cons.
> 2. If we go with the second option above, should we replace the countByKey / 
> count operators with aggregate(COUNT) as well? Personally I (Guozhang) feel 
> it is not necessary, as COUNT is a special aggregate function since we do not 
> need to map on any value fields; this is the same approach as in Spark as 
> well, where Count is built-in as first-citizen in the DSL, and others are 
> built-in as {{aggregate(SUM)}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska resolved KAFKA-3511.
-
Resolution: Won't Fix

> Add common aggregation functions like Sum and Avg as build-ins in Kafka 
> Streams DSL
> ---
>
> Key: KAFKA-3511
> URL: https://issues.apache.org/jira/browse/KAFKA-3511
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: api
> Fix For: 0.10.1.0
>
>
> Currently we have the following aggregation APIs in the Streams DSL:
> {code}
> KStream.aggregateByKey(..)
> KStream.reduceByKey(..)
> KStream.countByKey(..)
> KTable.groupBy(...).aggregate(..)
> KTable.groupBy(...).reduce(..)
> KTable.groupBy(...).count(..)
> {code}
> And it is better to add common aggregation functions like Sum and Avg as 
> built-in into the Streams DSL. A few questions to ask though:
> 1. Should we add those built-in functions as, for example 
> {{KTable.groupBy(...).sum(...)} or {{KTable.groupBy(...).aggregate(SUM, 
> ...)}}. Please see the comments below for detailed pros and cons.
> 2. If we go with the second option above, should we replace the countByKey / 
> count operators with aggregate(COUNT) as well? Personally I (Guozhang) feel 
> it is not necessary, as COUNT is a special aggregate function since we do not 
> need to map on any value fields; this is the same approach as in Spark as 
> well, where Count is built-in as first-citizen in the DSL, and others are 
> built-in as {{aggregate(SUM)}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka broker crash - broker id then changed

[cs user]

Hi All,

We are running Kafka version 0.9.0.1, at the time the brokers crashed
yesterday we were running in a 2 mode cluster. This has now been increased
to 3.

We are not specifying a broker id and relying on kafka generating one.

After the brokers crashed (at exactly the same time) we left kafka stopped
for a while. After kafka was started back up, the broker id's on both
servers were incremented, they were 1001/1002 and they flipped to
1003/1004. This seemed to cause some problems as partitions were assigned
to broker id's which it believed had disappeared and so were not
recoverable.

We noticed that the broker id's are actually stored in:

/tmp/kafka-logs/meta.properties

So we set these back to what they were and restarted. Is there a reason why
these would change?

Below are the error logs from each server:

Server 1

[2016-05-25 09:05:52,827] INFO [ReplicaFetcherManager on broker 1002]
Removed fetcher for partitions [Topic1Heartbeat,1]
(kafka.server.ReplicaFetcherManager)
[2016-05-25 09:05:52,831] INFO Completed load of log Topic1Heartbeat-1 with
log end offset 0 (kafka.log.Log)
[2016-05-25 09:05:52,831] INFO Created log for partition
[Topic1Heartbeat,1] in /tmp/kafka-logs with properties {compression.type ->
producer, file.delete.delay.ms -> 6, max.message.bytes -> 112,
min.insync.replicas -> 1, segment.
jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5,
index.interval.bytes -> 4096, unclean.leader.election.enable -> true,
retention.bytes -> -1, delete.retention.ms -> 8640, cleanup.policy ->
delete, flush.ms -> 9
223372036854775807, segment.ms -> 60480, segment.bytes -> 1073741824,
retention.ms -> 60480, segment.index.bytes -> 10485760, flush.messages
-> 9223372036854775807}. (kafka.log.LogManager)
[2016-05-25 09:05:52,831] INFO Partition [Topic1Heartbeat,1] on broker
1002: No checkpointed highwatermark is found for partition
[Topic1Heartbeat,1] (kafka.cluster.Partition)
[2016-05-25 09:14:12,189] INFO [GroupCoordinator 1002]: Preparing to
restabilize group Topic1 with old generation 0
(kafka.coordinator.GroupCoordinator)
[2016-05-25 09:14:12,190] INFO [GroupCoordinator 1002]: Stabilized group
Topic1 generation 1 (kafka.coordinator.GroupCoordinator)
[2016-05-25 09:14:12,195] INFO [GroupCoordinator 1002]: Assignment received
from leader for group Topic1 for generation 1
(kafka.coordinator.GroupCoordinator)
[2016-05-25 09:14:12,749] FATAL [Replica Manager on Broker 1002]: Halting
due to unrecoverable I/O error while handling produce request:
 (kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log
'__consumer_offsets-0'
at kafka.log.Log.append(Log.scala:318)
at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:442)
at kafka.cluster.Partition$$anonfun$9.apply(Partition.scala:428)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:268)
at
kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:428)
at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:401)
at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:386)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:386)
at
kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:322)
at
kafka.coordinator.GroupMetadataManager.store(GroupMetadataManager.scala:228)
at
kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429)
at
kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:429)
at scala.Option.foreach(Option.scala:257)
at
kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:429)
at
kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:280)
at kafka.server.KafkaApis.handle(KafkaApis.scala:76)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException:
/tmp/kafka-logs/__consumer_offsets-0/.index (No such
file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at
kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:277)
at

[jira] [Commented] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301710#comment-15301710
 ] 

ASF GitHub Bot commented on KAFKA-3511:
---

Github user enothereska closed the pull request at:

https://github.com/apache/kafka/pull/1424


> Add common aggregation functions like Sum and Avg as build-ins in Kafka 
> Streams DSL
> ---
>
> Key: KAFKA-3511
> URL: https://issues.apache.org/jira/browse/KAFKA-3511
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: api
> Fix For: 0.10.1.0
>
>
> Currently we have the following aggregation APIs in the Streams DSL:
> {code}
> KStream.aggregateByKey(..)
> KStream.reduceByKey(..)
> KStream.countByKey(..)
> KTable.groupBy(...).aggregate(..)
> KTable.groupBy(...).reduce(..)
> KTable.groupBy(...).count(..)
> {code}
> And it is better to add common aggregation functions like Sum and Avg as 
> built-in into the Streams DSL. A few questions to ask though:
> 1. Should we add those built-in functions as, for example 
> {{KTable.groupBy(...).sum(...)} or {{KTable.groupBy(...).aggregate(SUM, 
> ...)}}. Please see the comments below for detailed pros and cons.
> 2. If we go with the second option above, should we replace the countByKey / 
> count operators with aggregate(COUNT) as well? Personally I (Guozhang) feel 
> it is not necessary, as COUNT is a special aggregate function since we do not 
> need to map on any value fields; this is the same approach as in Spark as 
> well, where Count is built-in as first-citizen in the DSL, and others are 
> built-in as {{aggregate(SUM)}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3511: Initial commit for aggregators [Wi...

[enothereska]

Github user enothereska closed the pull request at:

https://github.com/apache/kafka/pull/1424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---