[jira] [Created] (KAFKA-2288) Follow-up to KAFKA-2249 - reduce logging and testing
Gwen Shapira created KAFKA-2288: --- Summary: Follow-up to KAFKA-2249 - reduce logging and testing Key: KAFKA-2288 URL: https://issues.apache.org/jira/browse/KAFKA-2288 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira As [~junrao] commented on KAFKA-2249, we have a needless test and we are logging configuration for every single partition now. Lets reduce the noise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-2288) Follow-up to KAFKA-2249 - reduce logging and testing
[ https://issues.apache.org/jira/browse/KAFKA-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira reassigned KAFKA-2288: --- Assignee: Gwen Shapira Follow-up to KAFKA-2249 - reduce logging and testing Key: KAFKA-2288 URL: https://issues.apache.org/jira/browse/KAFKA-2288 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira As [~junrao] commented on KAFKA-2249, we have a needless test and we are logging configuration for every single partition now. Lets reduce the noise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2291) Documentation Error
UTKARSH BHATNAGAR created KAFKA-2291: Summary: Documentation Error Key: KAFKA-2291 URL: https://issues.apache.org/jira/browse/KAFKA-2291 Project: Kafka Issue Type: Bug Components: website Reporter: UTKARSH BHATNAGAR Priority: Trivial http://kafka.apache.org/documentation.html#design Under the design section, the last second paragraph says this Finally in cases where the stream is fed into other data systems for serving we new[knew] the system would have to be able to guarantee fault-tolerance in the presence of machine failures. But there is a spelling mistake, it should be knew instead of new. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export
I think BikeShed will be a great name. Can you clarify the scope? The KIP discusses a framework and also few examples for connectors. Does the addition include just the framework (and perhaps an example or two), or do we plan to start accepting connectors to Apache Kafka project? Gwen On Thu, Jun 18, 2015 at 3:09 PM, Jay Kreps j...@confluent.io wrote: I think the only problem we came up with was that Kafka KopyKat abbreviates as KKK which is not ideal in the US. Copykat would still be googlable without that issue. :-) -Jay On Thu, Jun 18, 2015 at 1:20 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Just a comment on the name. KopyKat? More unique, easy to write, pronounce, remember... Otis On Jun 18, 2015, at 13:36, Jay Kreps j...@confluent.io wrote: 1. We were calling the plugins connectors (which is kind of a generic way to say either source or sink) and the framework copycat. The pro of copycat is it is kind of fun. The con is that it doesn't really say what it does. The Kafka Connector Framework would be a duller but more intuitive name, but I suspect people would then just shorten it to KCF which again has no intuitive meaning. 2. Potentially. One alternative we had thought of wrt the consumer was to have the protocol just handle the group management part and have the partition assignment be purely driven by the client. At the time copycat wasn't even a twinkle in our eyes so we weren't really thinking about that. There were pros and cons to this and we decided it was better to do partition assignment on the broker side. We could revisit this, it might not be a massive change in the consumer, but it would definitely add work there. I do agree that if we have learned one thing it is to keep clients away from zk. This zk usage is more limited though, in that there is no intention of having copycat in different languages as the clients are. 4. I think the idea is to include the structural schema information you have available so it can be taken advantage of. Obviously the easiest approach would just be to have a static schema for the messages like timestamp + string/byte[]. However this means that i the source has schema information there is no real official way to propagate that. Having a real built-in schema mechanism gives you a little more power to make the data usable. So if you were publishing apache logs the low-touch generic way would just be to have the schema be string since that is what apache log entries are. However if you had the actual format string used for the log you could use that to have a richer schema and parse out the individual fields, which is significantly more usable. The advantage of this is that systems like databases, Hadoop, and so on that have some notion of schemas can take advantage of this information that is captured with the source data. So, e.g. the JDBC plugin can map the individual fields to columns automatically, and you can support features like projecting out particular fields and renaming fields easily without having to write custom source-specific code. -Jay On Tue, Jun 16, 2015 at 5:00 PM, Joe Stein joe.st...@stealth.ly wrote: Hey Ewen, very interesting! I like the idea of the connector and making one side always being Kafka for all the reasons you mentioned. It makes having to build consumers (over and over and over (and over)) again for these type of tasks much more consistent for everyone. Some initial comments (will read a few more times and think more through it). 1) Copycat, it might be weird/hard to talk about producers, consumers, brokers and copycat for what and how kafka runs. I think the other naming makes sense but maybe we can call it something else? Sinks or whatever (don't really care just bringing up it might be something to consider). We could also just call it connectors...dunno producers, consumers, brokers and connectors... 2) Can we do copycat-workers without having to rely on Zookeeper? So much work has been done to remove this dependency if we can do something without ZK lets try (or at least abstract it so it is easier later to make it pluggable). 3) Even though connectors being managed in project has already been rejected... maybe we want to have a few (or one) that are in the project and maintained. This makes out of the box really out of the box (if only file or hdfs or something). 4) all records include schemas which describe the format of their data I don't totally get this... a lot of data doesn't have the schema with it, we have to plug that in... so would the plugin you are talking about for serializer would inject the schema to use with the record when it sees the data? ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Jun 16, 2015 at 4:33 PM, Ewen
[GitHub] kafka pull request: KAFKA-2289: KafkaProducer logs erroneous warni...
GitHub user hgschmie opened a pull request: https://github.com/apache/kafka/pull/71 KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hgschmie/kafka KAFKA-2289 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit ef6c0961c905d65ff8997a1ba7a09ded617b9837 Author: Henning Schmiedehausen henn...@groupon.com Date: 2015-06-19T23:17:38Z KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Build failed in Jenkins: Kafka-trunk #515
See https://builds.apache.org/job/Kafka-trunk/515/changes Changes: [junrao] kafka-2265; creating a topic with large number of partitions takes a long time; patched by Manikumar Reddy; reviewed by Jun Rao [junrao] kafka-2234; Partition reassignment of a nonexistent topic prevents future reassignments; patched by Manikumar Reddy; reviewed by Jun Rao [junrao] trivial change to fix unit test failure introduced in kafka-2234 [junrao] kafka-1758; corrupt recovery file prevents startup; patched by Manikumar Reddy; reviewed by Neha Narkhede and Jun Rao [junrao] kafka-1646; Improve consumer read performance for Windows; patched by Honghai Chen; reviewed by Jay Kreps and Jun Rao [junrao] kafka-2012; Broker should automatically handle corrupt index files; patched by Manikumar Reddy; reviewed by Jun Rao -- [...truncated 772 lines...] kafka.log.LogConfigTest testFromPropsEmpty PASSED kafka.log.LogConfigTest testFromPropsToProps PASSED kafka.log.LogConfigTest testFromPropsInvalid PASSED kafka.log.OffsetMapTest testBasicValidation PASSED kafka.log.OffsetMapTest testClear PASSED kafka.log.LogManagerTest testCreateLog PASSED kafka.log.LogManagerTest testGetNonExistentLog PASSED kafka.log.LogManagerTest testCleanupExpiredSegments PASSED kafka.log.LogManagerTest testCleanupSegmentsToMaintainSize PASSED kafka.log.LogManagerTest testTimeBasedFlush PASSED kafka.log.LogManagerTest testLeastLoadedAssignment PASSED kafka.log.LogManagerTest testTwoLogManagersUsingSameDirFails PASSED kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithTrailingSlash PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[1] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[2] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[3] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[4] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[5] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[6] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[7] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[8] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[9] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[10] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[11] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[12] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[13] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[14] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[15] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[16] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[17] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[18] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[19] PASSED kafka.log.LogTest testTimeBasedLogRoll PASSED kafka.log.LogTest testTimeBasedLogRollJitter PASSED kafka.log.LogTest testSizeBasedLogRoll PASSED kafka.log.LogTest testLoadEmptyLog PASSED kafka.log.LogTest testAppendAndReadWithSequentialOffsets PASSED kafka.log.LogTest testAppendAndReadWithNonSequentialOffsets PASSED kafka.log.LogTest testReadAtLogGap PASSED kafka.log.LogTest testReadOutOfRange PASSED kafka.log.LogTest testLogRolls PASSED kafka.log.LogTest testCompressedMessages PASSED kafka.log.LogTest testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED kafka.log.LogTest testMessageSetSizeCheck PASSED kafka.log.LogTest testCompactedTopicConstraints PASSED kafka.log.LogTest testMessageSizeCheck PASSED kafka.log.LogTest testLogRecoversToCorrectOffset PASSED kafka.log.LogTest testIndexRebuild PASSED kafka.log.LogTest testCorruptIndexRebuild PASSED kafka.log.LogTest testTruncateTo PASSED kafka.log.LogTest testIndexResizingAtTruncation PASSED kafka.log.LogTest testBogusIndexSegmentsAreRemoved PASSED kafka.log.LogTest testReopenThenTruncate PASSED kafka.log.LogTest testAsyncDelete PASSED kafka.log.LogTest testOpenDeletesObsoleteFiles PASSED kafka.log.LogTest testAppendMessageWithNullPayload PASSED kafka.log.LogTest testCorruptLog PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.log.LogCleanerIntegrationTest
Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export
I think we want the connectors to be federated just because trying to maintain all the connectors centrally would be really painful. I think if we really do this well we would want to have 100 of these connectors so it really won't make sense to maintain them with the project. I think the thought was just to include the framework and maybe one simple connector as an example. Thoughts? -Jay On Fri, Jun 19, 2015 at 2:38 PM, Gwen Shapira gshap...@cloudera.com wrote: I think BikeShed will be a great name. Can you clarify the scope? The KIP discusses a framework and also few examples for connectors. Does the addition include just the framework (and perhaps an example or two), or do we plan to start accepting connectors to Apache Kafka project? Gwen On Thu, Jun 18, 2015 at 3:09 PM, Jay Kreps j...@confluent.io wrote: I think the only problem we came up with was that Kafka KopyKat abbreviates as KKK which is not ideal in the US. Copykat would still be googlable without that issue. :-) -Jay On Thu, Jun 18, 2015 at 1:20 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Just a comment on the name. KopyKat? More unique, easy to write, pronounce, remember... Otis On Jun 18, 2015, at 13:36, Jay Kreps j...@confluent.io wrote: 1. We were calling the plugins connectors (which is kind of a generic way to say either source or sink) and the framework copycat. The pro of copycat is it is kind of fun. The con is that it doesn't really say what it does. The Kafka Connector Framework would be a duller but more intuitive name, but I suspect people would then just shorten it to KCF which again has no intuitive meaning. 2. Potentially. One alternative we had thought of wrt the consumer was to have the protocol just handle the group management part and have the partition assignment be purely driven by the client. At the time copycat wasn't even a twinkle in our eyes so we weren't really thinking about that. There were pros and cons to this and we decided it was better to do partition assignment on the broker side. We could revisit this, it might not be a massive change in the consumer, but it would definitely add work there. I do agree that if we have learned one thing it is to keep clients away from zk. This zk usage is more limited though, in that there is no intention of having copycat in different languages as the clients are. 4. I think the idea is to include the structural schema information you have available so it can be taken advantage of. Obviously the easiest approach would just be to have a static schema for the messages like timestamp + string/byte[]. However this means that i the source has schema information there is no real official way to propagate that. Having a real built-in schema mechanism gives you a little more power to make the data usable. So if you were publishing apache logs the low-touch generic way would just be to have the schema be string since that is what apache log entries are. However if you had the actual format string used for the log you could use that to have a richer schema and parse out the individual fields, which is significantly more usable. The advantage of this is that systems like databases, Hadoop, and so on that have some notion of schemas can take advantage of this information that is captured with the source data. So, e.g. the JDBC plugin can map the individual fields to columns automatically, and you can support features like projecting out particular fields and renaming fields easily without having to write custom source-specific code. -Jay On Tue, Jun 16, 2015 at 5:00 PM, Joe Stein joe.st...@stealth.ly wrote: Hey Ewen, very interesting! I like the idea of the connector and making one side always being Kafka for all the reasons you mentioned. It makes having to build consumers (over and over and over (and over)) again for these type of tasks much more consistent for everyone. Some initial comments (will read a few more times and think more through it). 1) Copycat, it might be weird/hard to talk about producers, consumers, brokers and copycat for what and how kafka runs. I think the other naming makes sense but maybe we can call it something else? Sinks or whatever (don't really care just bringing up it might be something to consider). We could also just call it connectors...dunno producers, consumers, brokers and connectors... 2) Can we do copycat-workers without having to rely on Zookeeper? So much work has been done to remove this dependency if we can do something without ZK lets try (or at least abstract it so it is easier later to make it pluggable). 3) Even though connectors being managed in project has already been rejected... maybe we want to have a few (or one) that are
[jira] [Resolved] (KAFKA-2286) Cannot produce messages remotely: No partition metadata for topic test due to kafka.common.LeaderNotAvailableException
[
https://issues.apache.org/jira/browse/KAFKA-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Bridge resolved KAFKA-2286.
Resolution: Not A Problem
Fix Version/s: 0.8.1.2
This turns out to be a problem with my configuration. it was a
firewall/routing issue, but between zookeeper and the broker on the same
machine, not between my client and the machine hosting zookeeper and the
broker. Zookeeper uses the external IP address it receives from the
advertised.host.name to connect back to the broker. So everything worked when
it was not set---I think it must default to localhost---but it the two couldn't
communicate when it was connecting to the broker on same machine via its
external IP.
I fixed this with a route back so that it goes back through NAT to itself.
It's probably a pretty obscure situation, but it might help to have a clearer
error when the zookeeper can't contact the broker.
Cannot produce messages remotely: No partition metadata for topic test due to
kafka.common.LeaderNotAvailableException
--
Key: KAFKA-2286
URL: https://issues.apache.org/jira/browse/KAFKA-2286
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.1
Environment: CentOS 7 / Oracle jdk1.8.0_45
Reporter: Mike Bridge
Assignee: Jun Rao
Priority: Critical
Fix For: 0.8.1.2
I successfully ran the QuickStart tutorial locally on a single EC2 machine.
I stopped everything running on the server, deleted the logs in /tmp, and
added the single line advertised.host.name=MY.EXTERNAL.IP to
server.properties. (Port 9092 is indeed allowed through the firewall and
responding, so no problem there.)
I then restarted the server and successfully created a topic as per the
QuickStart tutorial.
Then, I ran the producer in exactly the same way as in the QuickStart
tutorial, except on a remote machine:
bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092 --topic
test
This generates the following error:
WARN Error while fetching metadata [{TopicMetadata for topic test123 -
No partition metadata for topic test123 due to
kafka.common.LeaderNotAvailableException}] for topic [test123]: class
kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
See also:
http://stackoverflow.com/questions/30880811/kafka-quickstart-advertised-host-name-gives-kafka-common-leadernotavailableexce/30919109#30919109
I've spent several evenings trying to get this simple Kafka setup to work but
am stuck. It seems like it should work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-2289) KafkaProducer logs erroneous warning on startu.
Henning Schmiedehausen created KAFKA-2289: - Summary: KafkaProducer logs erroneous warning on startu. Key: KAFKA-2289 URL: https://issues.apache.org/jira/browse/KAFKA-2289 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Henning Schmiedehausen Priority: Trivial When creating a new KafkaProducer using the KafkaProducer(KafkaConfig, SerializerK, SerializerV) constructor, Kafka will list the following lines, which are harmless but are still at WARN level: WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration value.serializer = class was supplied but isn't a known config. WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration key.serializer = class was supplied but isn't a known config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2289) KafkaProducer logs erroneous warning on startup
[ https://issues.apache.org/jira/browse/KAFKA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henning Schmiedehausen updated KAFKA-2289: -- Summary: KafkaProducer logs erroneous warning on startup (was: KafkaProducer logs erroneous warning on startu.) KafkaProducer logs erroneous warning on startup --- Key: KAFKA-2289 URL: https://issues.apache.org/jira/browse/KAFKA-2289 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Henning Schmiedehausen Priority: Trivial When creating a new KafkaProducer using the KafkaProducer(KafkaConfig, SerializerK, SerializerV) constructor, Kafka will list the following lines, which are harmless but are still at WARN level: WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration value.serializer = class was supplied but isn't a known config. WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration key.serializer = class was supplied but isn't a known config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2290) OffsetIndex should open RandomAccessFile consistently
Jun Rao created KAFKA-2290: -- Summary: OffsetIndex should open RandomAccessFile consistently Key: KAFKA-2290 URL: https://issues.apache.org/jira/browse/KAFKA-2290 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao We open RandomAccessFile in rw mode in the constructor, but in rws mode in resize(). We should use rw in both cases since it's more efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: KafkaPreCommit #127
See https://builds.apache.org/job/KafkaPreCommit/127/changes Changes: [junrao] kafka-2234; Partition reassignment of a nonexistent topic prevents future reassignments; patched by Manikumar Reddy; reviewed by Jun Rao [junrao] trivial change to fix unit test failure introduced in kafka-2234 [junrao] kafka-1758; corrupt recovery file prevents startup; patched by Manikumar Reddy; reviewed by Neha Narkhede and Jun Rao [junrao] kafka-1646; Improve consumer read performance for Windows; patched by Honghai Chen; reviewed by Jay Kreps and Jun Rao [junrao] kafka-2012; Broker should automatically handle corrupt index files; patched by Manikumar Reddy; reviewed by Jun Rao -- [...truncated 1383 lines...] kafka.log.LogConfigTest testFromPropsEmpty PASSED kafka.log.LogConfigTest testFromPropsToProps PASSED kafka.log.LogConfigTest testFromPropsInvalid PASSED kafka.log.OffsetMapTest testBasicValidation PASSED kafka.log.OffsetMapTest testClear PASSED kafka.log.LogManagerTest testCreateLog PASSED kafka.log.LogManagerTest testGetNonExistentLog PASSED kafka.log.LogManagerTest testCleanupExpiredSegments PASSED kafka.log.LogManagerTest testCleanupSegmentsToMaintainSize PASSED kafka.log.LogManagerTest testTimeBasedFlush PASSED kafka.log.LogManagerTest testLeastLoadedAssignment PASSED kafka.log.LogManagerTest testTwoLogManagersUsingSameDirFails PASSED kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithTrailingSlash PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[1] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[2] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[3] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[4] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[5] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[6] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[7] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[8] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[9] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[10] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[11] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[12] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[13] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[14] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[15] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[16] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[17] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[18] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[19] PASSED kafka.log.LogTest testTimeBasedLogRoll PASSED kafka.log.LogTest testTimeBasedLogRollJitter PASSED kafka.log.LogTest testSizeBasedLogRoll PASSED kafka.log.LogTest testLoadEmptyLog PASSED kafka.log.LogTest testAppendAndReadWithSequentialOffsets PASSED kafka.log.LogTest testAppendAndReadWithNonSequentialOffsets PASSED kafka.log.LogTest testReadAtLogGap PASSED kafka.log.LogTest testReadOutOfRange PASSED kafka.log.LogTest testLogRolls PASSED kafka.log.LogTest testCompressedMessages PASSED kafka.log.LogTest testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED kafka.log.LogTest testMessageSetSizeCheck PASSED kafka.log.LogTest testCompactedTopicConstraints PASSED kafka.log.LogTest testMessageSizeCheck PASSED kafka.log.LogTest testLogRecoversToCorrectOffset PASSED kafka.log.LogTest testIndexRebuild PASSED kafka.log.LogTest testCorruptIndexRebuild PASSED kafka.log.LogTest testTruncateTo PASSED kafka.log.LogTest testIndexResizingAtTruncation PASSED kafka.log.LogTest testBogusIndexSegmentsAreRemoved PASSED kafka.log.LogTest testReopenThenTruncate PASSED kafka.log.LogTest testAsyncDelete PASSED kafka.log.LogTest testOpenDeletesObsoleteFiles PASSED kafka.log.LogTest testAppendMessageWithNullPayload PASSED kafka.log.LogTest testCorruptLog PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[0] PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[1] PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[2]
[jira] [Commented] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593953#comment-14593953 ] Jun Rao commented on KAFKA-2012: I am not sure if it's better to add another config. Perhaps, we can just save the corrupted file as Gwen suggested in KAFKA-1554 for trouble shooting. Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Fix For: 0.8.3 Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2289) KafkaProducer logs erroneous warning on startup
[ https://issues.apache.org/jira/browse/KAFKA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594087#comment-14594087 ] ASF GitHub Bot commented on KAFKA-2289: --- GitHub user hgschmie opened a pull request: https://github.com/apache/kafka/pull/71 KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hgschmie/kafka KAFKA-2289 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit ef6c0961c905d65ff8997a1ba7a09ded617b9837 Author: Henning Schmiedehausen henn...@groupon.com Date: 2015-06-19T23:17:38Z KAFKA-2289: KafkaProducer logs erroneous warning on startup This change fixes the problem. KafkaProducer logs erroneous warning on startup --- Key: KAFKA-2289 URL: https://issues.apache.org/jira/browse/KAFKA-2289 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Henning Schmiedehausen Priority: Trivial When creating a new KafkaProducer using the KafkaProducer(KafkaConfig, SerializerK, SerializerV) constructor, Kafka will list the following lines, which are harmless but are still at WARN level: WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration value.serializer = class was supplied but isn't a known config. WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration key.serializer = class was supplied but isn't a known config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2289) KafkaProducer logs erroneous warning on startup
[ https://issues.apache.org/jira/browse/KAFKA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594089#comment-14594089 ] Henning Schmiedehausen commented on KAFKA-2289: --- Pull request with a fix is at https://github.com/apache/kafka/pull/71 KafkaProducer logs erroneous warning on startup --- Key: KAFKA-2289 URL: https://issues.apache.org/jira/browse/KAFKA-2289 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Reporter: Henning Schmiedehausen Priority: Trivial When creating a new KafkaProducer using the KafkaProducer(KafkaConfig, SerializerK, SerializerV) constructor, Kafka will list the following lines, which are harmless but are still at WARN level: WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration value.serializer = class was supplied but isn't a known config. WARN [2015-06-19 23:13:56,557] org.apache.kafka.clients.producer.ProducerConfig: The configuration key.serializer = class was supplied but isn't a known config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export
My initial thoughts: Although it is kind of discussed very broadly, I did struggle a bit to properly grasp the value add this adds over the alternative approaches that are available today (or need a little work to accomplish) in specific use cases. I feel its better to take specific common use cases and show why this will do better to make it clear. For example data flow starting from a pool of web server and finally end up in HDFS or Hive while providing At-least one guarantees. Below are more specific points that occurred to me: - Import: Today we can create data flows to pick up data from a variety of source and push data into Kafka using Flume. Not clear how this system can do better in this specific case. - Export: For pulling data out of Kakfa there is Camus (which limits destination to HDFS), Flume (which can deliver to many places) and also Sqoop (which could be extended to support Kafka). Camus and Sqoop don't have the problem of requires defining many tasks issue for parallelism. - YARN support – Letting YARN manage things is actually good thing (not a bad thing as indicated), since its easier for the scaling in/out as needed and not worry too much about hardware allocation. - Exactly-Once: It is clear that on the import side you won't support that for now. Not clear how you will support that on export side for destination like HDFS or some other. Exactly once only make sense when we can have that guarantee on the entire data flow (not just portions of the flow). - Connector Variety: Flume and Sqoop already have out of the box- support for multiple destinations and sources. -roshan On 6/19/15 2:47 PM, Jay Kreps j...@confluent.iomailto:j...@confluent.io wrote: I think we want the connectors to be federated just because trying to maintain all the connectors centrally would be really painful. I think if we really do this well we would want to have 100 of these connectors so it really won't make sense to maintain them with the project. I think the thought was just to include the framework and maybe one simple connector as an example. Thoughts? -Jay On Fri, Jun 19, 2015 at 2:38 PM, Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com wrote: I think BikeShed will be a great name. Can you clarify the scope? The KIP discusses a framework and also few examples for connectors. Does the addition include just the framework (and perhaps an example or two), or do we plan to start accepting connectors to Apache Kafka project? Gwen On Thu, Jun 18, 2015 at 3:09 PM, Jay Kreps j...@confluent.iomailto:j...@confluent.io wrote: I think the only problem we came up with was that Kafka KopyKat abbreviates as KKK which is not ideal in the US. Copykat would still be googlable without that issue. :-) -Jay On Thu, Jun 18, 2015 at 1:20 PM, Otis Gospodnetic otis.gospodne...@gmail.commailto:otis.gospodne...@gmail.com wrote: Just a comment on the name. KopyKat? More unique, easy to write, pronounce, remember... Otis On Jun 18, 2015, at 13:36, Jay Kreps j...@confluent.iomailto:j...@confluent.io wrote: 1. We were calling the plugins connectors (which is kind of a generic way to say either source or sink) and the framework copycat. The pro of copycat is it is kind of fun. The con is that it doesn't really say what it does. The Kafka Connector Framework would be a duller but more intuitive name, but I suspect people would then just shorten it to KCF which again has no intuitive meaning. 2. Potentially. One alternative we had thought of wrt the consumer was to have the protocol just handle the group management part and have the partition assignment be purely driven by the client. At the time copycat wasn't even a twinkle in our eyes so we weren't really thinking about that. There were pros and cons to this and we decided it was better to do partition assignment on the broker side. We could revisit this, it might not be a massive change in the consumer, but it would definitely add work there. I do agree that if we have learned one thing it is to keep clients away from zk. This zk usage is more limited though, in that there is no intention of having copycat in different languages as the clients are. 4. I think the idea is to include the structural schema information you have available so it can be taken advantage of. Obviously the easiest approach would just be to have a static schema for the messages like timestamp + string/byte[]. However this means that i the source has schema information there is no real official way to propagate that. Having a real built-in schema mechanism gives you a little more power to make the data usable. So if you were publishing apache logs the low-touch generic way would just be to have the schema be string since that is what apache log entries are. However if you had the actual format string used for the log you could use that to have a richer schema and parse out the
Review Request 35677: Patch for KAFKA-2288
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35677/ --- Review request for kafka. Bugs: KAFKA-2288 https://issues.apache.org/jira/browse/KAFKA-2288 Repository: kafka Description --- minor corrections to LogConfig and KafkaConfigTest Diffs - clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java bae528d31516679bed88ee61b408f209f185a8cc core/src/main/scala/kafka/log/LogConfig.scala fc41132d2bf29439225ec581829eb479f98cc416 core/src/main/scala/kafka/server/KafkaServer.scala 52dc728bb1ab4b05e94dc528da1006040e2f28c9 core/src/test/scala/unit/kafka/log/LogConfigTest.scala 19dcb47f3f406b8d6c3668297450ab6b534e4471 core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 98a5b042a710d3c1064b0379db1d152efc9eabee core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 2428dbd7197a58cf4cad42ef82b385dab3a2b15e Diff: https://reviews.apache.org/r/35677/diff/ Testing --- Thanks, Gwen Shapira
[jira] [Commented] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594420#comment-14594420 ] Manikumar Reddy commented on KAFKA-2012: I am not sure if we can find out the reason after file corruption. i,e What we can infer from a corrupted file? instead we should add some defensive code/logs before and after index reads/writes. looks like KAFKA-1554 got some steps to reproduce the issue. Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Fix For: 0.8.3 Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2245) Add response tests for ConsumerCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-2245: Attachment: KAFKA-2245_2015-06-19_22:55:56.patch Add response tests for ConsumerCoordinator -- Key: KAFKA-2245 URL: https://issues.apache.org/jira/browse/KAFKA-2245 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Onur Karaman Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-2245.patch, KAFKA-2245_2015-06-19_22:55:56.patch We can validate error codes from JoinGroupResponses and HeartbeatResponses. Currently this includes: - JoinGroupRequest to the wrong coordinator returns NOT_COORDINATOR_FOR_CONSUMER - JoinGroupRequest with an unknown partition assignment strategy returns UNKNOWN_PARTITION_ASSIGNMENT_STRATEGY - JoinGroupRequest with an out-of-range session timeout returns INVALID_SESSION_TIMEOUT - JoinGroupRequest on a brand new group with an unrecognized consumer id produces UNKNOWN_CONSUMER_ID - JoinGroupRequest with mismatched partition assignment strategy compared to the rest of the group returns INCONSISTENT_PARTITION_ASSIGNMENT_STRATEGY - JoinGroupRequest on an existing group with an unrecognized consumer id produces UNKNOWN_CONSUMER_ID - A correct JoinGroupRequest returns NONE - HeartbeatRequest to the wrong coordinator returns NOT_COORDINATOR_FOR_CONSUMER - HeartbeatRequest with an unknown group returns UNKNOWN_CONSUMER_ID - HeartbeatRequest with an unrecognized consumer id returns UNKNOWN_CONSUMER_ID - HeartbeatRequest with generation id mismatch returns ILLEGAL_GENERATION - A correct HeartbeatRequest returns NONE We can validate the generation id increments on rebalance based on the JoinGroupResponse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2245) Add response tests for ConsumerCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594421#comment-14594421 ] Onur Karaman commented on KAFKA-2245: - Updated reviewboard https://reviews.apache.org/r/35086/diff/ against branch origin/trunk Add response tests for ConsumerCoordinator -- Key: KAFKA-2245 URL: https://issues.apache.org/jira/browse/KAFKA-2245 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Onur Karaman Assignee: Onur Karaman Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-2245.patch, KAFKA-2245_2015-06-19_22:55:56.patch We can validate error codes from JoinGroupResponses and HeartbeatResponses. Currently this includes: - JoinGroupRequest to the wrong coordinator returns NOT_COORDINATOR_FOR_CONSUMER - JoinGroupRequest with an unknown partition assignment strategy returns UNKNOWN_PARTITION_ASSIGNMENT_STRATEGY - JoinGroupRequest with an out-of-range session timeout returns INVALID_SESSION_TIMEOUT - JoinGroupRequest on a brand new group with an unrecognized consumer id produces UNKNOWN_CONSUMER_ID - JoinGroupRequest with mismatched partition assignment strategy compared to the rest of the group returns INCONSISTENT_PARTITION_ASSIGNMENT_STRATEGY - JoinGroupRequest on an existing group with an unrecognized consumer id produces UNKNOWN_CONSUMER_ID - A correct JoinGroupRequest returns NONE - HeartbeatRequest to the wrong coordinator returns NOT_COORDINATOR_FOR_CONSUMER - HeartbeatRequest with an unknown group returns UNKNOWN_CONSUMER_ID - HeartbeatRequest with an unrecognized consumer id returns UNKNOWN_CONSUMER_ID - HeartbeatRequest with generation id mismatch returns ILLEGAL_GENERATION - A correct HeartbeatRequest returns NONE We can validate the generation id increments on rebalance based on the JoinGroupResponse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35086: add response tests for ConsumerCoordinator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35086/ --- (Updated June 20, 2015, 5:56 a.m.) Review request for kafka. Bugs: KAFKA-2245 https://issues.apache.org/jira/browse/KAFKA-2245 Repository: kafka Description (updated) --- coordinator response test Diffs (updated) - core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 51e89c87ee2c20fc7f976536f01fa1055fb8e670 core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala c39e6de34ee531c6dfa9107b830752bd7f8fbe59 core/src/test/scala/unit/kafka/coordinator/ConsumerCoordinatorResponseTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/35086/diff/ Testing --- Thanks, Onur Karaman
[jira] [Updated] (KAFKA-2288) Follow-up to KAFKA-2249 - reduce logging and testing
[ https://issues.apache.org/jira/browse/KAFKA-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2288: Attachment: KAFKA-2288.patch Follow-up to KAFKA-2249 - reduce logging and testing Key: KAFKA-2288 URL: https://issues.apache.org/jira/browse/KAFKA-2288 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-2288.patch As [~junrao] commented on KAFKA-2249, we have a needless test and we are logging configuration for every single partition now. Lets reduce the noise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2288) Follow-up to KAFKA-2249 - reduce logging and testing
[ https://issues.apache.org/jira/browse/KAFKA-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2288: Status: Patch Available (was: Open) Follow-up to KAFKA-2249 - reduce logging and testing Key: KAFKA-2288 URL: https://issues.apache.org/jira/browse/KAFKA-2288 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-2288.patch As [~junrao] commented on KAFKA-2249, we have a needless test and we are logging configuration for every single partition now. Lets reduce the noise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2288) Follow-up to KAFKA-2249 - reduce logging and testing
[ https://issues.apache.org/jira/browse/KAFKA-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594222#comment-14594222 ] Gwen Shapira commented on KAFKA-2288: - Created reviewboard https://reviews.apache.org/r/35677/diff/ against branch trunk Follow-up to KAFKA-2249 - reduce logging and testing Key: KAFKA-2288 URL: https://issues.apache.org/jira/browse/KAFKA-2288 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Attachments: KAFKA-2288.patch As [~junrao] commented on KAFKA-2249, we have a needless test and we are logging configuration for every single partition now. Lets reduce the noise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Honghai Chen updated KAFKA-1646: Attachment: (was: KAFKA-1646_20150618_225612.patch) Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch, KAFKA-1646_20150616_FixFormat.patch, KAFKA-1646_20150618_235231.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-1465) kafka-reassign-partitions.sh fails when topic name contains dash/hyphen
[
https://issues.apache.org/jira/browse/KAFKA-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar Reddy resolved KAFKA-1465.
Resolution: Fixed
Fix Version/s: 0.8.3
Not able to reproduce this issue on trunk. Might have fixed in previous
commits. So closing this issue.
kafka-reassign-partitions.sh fails when topic name contains dash/hyphen
---
Key: KAFKA-1465
URL: https://issues.apache.org/jira/browse/KAFKA-1465
Project: Kafka
Issue Type: Bug
Components: tools
Affects Versions: 0.8.0
Reporter: Adrian Muraru
Fix For: 0.8.3
{{./bin/kafka-reassign-partitions.sh --topics-to-move-json-file
~/rebalance-topic.json --broker-list 18,19 --zookeeper $ZK_QUORUM
--execute}}
{code}
Partitions reassignment failed due to Can't parse json string: null
kafka.common.KafkaException: Can't parse json string: null
at kafka.utils.Json$.liftedTree1$1(Json.scala:36)
at kafka.utils.Json$.parseFull(Json.scala:32)
at
kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:529)
at
kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:525)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at kafka.utils.ZkUtils$.getReplicaAssignmentForTopics(ZkUtils.scala:525)
at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:112)
at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
Caused by: java.lang.NullPointerException
at
scala.util.parsing.combinator.lexical.Scanners$Scanner.init(Scanners.scala:52)
at scala.util.parsing.json.JSON$.parseRaw(JSON.scala:71)
at scala.util.parsing.json.JSON$.parseFull(JSON.scala:85)
at kafka.utils.Json$.liftedTree1$1(Json.scala:33)
... 8 more
{code}
*rebalance-topic.json*
{code}
{topics:
[{topic: metrics-logs}],
version:1
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Honghai Chen updated KAFKA-1646: Attachment: KAFKA-1646_20150618_235231.patch Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch, KAFKA-1646_20150616_FixFormat.patch, KAFKA-1646_20150618_225612.patch, KAFKA-1646_20150618_235231.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593116#comment-14593116 ] Honghai Chen commented on KAFKA-1646: - Updated reviewboard https://reviews.apache.org/r/33204/diff/ against branch origin/trunk Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch, KAFKA-1646_20150616_FixFormat.patch, KAFKA-1646_20150618_225612.patch, KAFKA-1646_20150618_235231.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2012: --- Status: Patch Available (was: In Progress) Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2012: --- Attachment: KAFKA-2012_2015-06-19_18:55:11.patch Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35503: Patch for KAFKA-2012
On June 19, 2015, 12:11 a.m., Jun Rao wrote: core/src/main/scala/kafka/log/Log.scala, lines 225-227 https://reviews.apache.org/r/35503/diff/1/?file=985641#file985641line225 Do we still need this now that we have verified the index consistency earlier? Yes, we can remove these statements On June 19, 2015, 12:11 a.m., Jun Rao wrote: core/src/main/scala/kafka/log/Log.scala, line 178 https://reviews.apache.org/r/35503/diff/1/?file=985641#file985641line178 Is this correct? The mmapping of the index file happens in the initialization of OffsetIndex. It seems that we need to re-create the LogSegment after the index file is deleted. index file gets recreated in OffsetIndex.mmap() method (val newlyCreated = file.createNewFile()). So we no need to re-create the LogSegment. Pl check. - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35503/#review88470 --- On June 19, 2015, 1:27 p.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35503/ --- (Updated June 19, 2015, 1:27 p.m.) Review request for kafka. Bugs: KAFKA-2012 https://issues.apache.org/jira/browse/KAFKA-2012 Repository: kafka Description --- Address Jun's comments Diffs - core/src/main/scala/kafka/log/Log.scala 84e7b8fe9dd014884b60c4fbe13c835cf02a40e4 core/src/test/scala/unit/kafka/log/LogTest.scala a8e57c2348e694a1e0d5407ae8ae105fb1f1bf59 Diff: https://reviews.apache.org/r/35503/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Commented] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593418#comment-14593418 ] Manikumar Reddy commented on KAFKA-2012: Updated reviewboard https://reviews.apache.org/r/35503/diff/ against branch origin/trunk Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35503: Patch for KAFKA-2012
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35503/ --- (Updated June 19, 2015, 1:27 p.m.) Review request for kafka. Bugs: KAFKA-2012 https://issues.apache.org/jira/browse/KAFKA-2012 Repository: kafka Description (updated) --- Address Jun's comments Diffs (updated) - core/src/main/scala/kafka/log/Log.scala 84e7b8fe9dd014884b60c4fbe13c835cf02a40e4 core/src/test/scala/unit/kafka/log/LogTest.scala a8e57c2348e694a1e0d5407ae8ae105fb1f1bf59 Diff: https://reviews.apache.org/r/35503/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Resolved] (KAFKA-1656) replica-assignment option in bin/kakfa-topics.sh doesn't work with alter
[ https://issues.apache.org/jira/browse/KAFKA-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy resolved KAFKA-1656. Resolution: Duplicate This got fixed in KAFKA-1947 replica-assignment option in bin/kakfa-topics.sh doesn't work with alter --- Key: KAFKA-1656 URL: https://issues.apache.org/jira/browse/KAFKA-1656 Project: Kafka Issue Type: Improvement Components: tools Affects Versions: 0.8.1.1 Environment: all Reporter: jayden sun Priority: Minor Labels: newbie replica-assignment option in bin/kakfa-topics.sh works only with --create and doesn't work with --alter, but the config comment is somewhat confusing. For example: bin/kafka-topics.sh --create --zookeeper 10.0.44.137:2181 --replica-assignment 1:2,1:2 --partitions 2 --topic topic i use the command above to create a topic with two partitions on broker 1 and 2. and then i want to change partition count from two to four, and the newly created partition on broker 3 and 4. So i try the command: bin/kafka-topics.sh --alter --zookeeper 10.0.44.137:2181 --replica-assignment 3:4,3:4 --partitions 4 --topic topic But it does not works!!! replica-assignment can't be used with partitions. What i can do is to add partitions like this: bin/kafka-topics.sh --alter --zookeeper 10.0.44.137:2181 --partitions 4 --topic topic and then use the partition reassignment command to change the brokers for newly partitions to 3 and 4. So,i think or altered in the comment for --replica-assignment should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593116#comment-14593116 ] Honghai Chen edited comment on KAFKA-1646 at 6/19/15 6:57 AM: -- Updated reviewboard https://reviews.apache.org/r/33204/diff/ against branch origin/trunk The latest code review :https://reviews.apache.org/r/33204/diff/8/ 1, fix 2 test cases. 2, fix logCleaner.Whenever create one new file, should set the preallocate parameter. 3, merge to latest trunk. Is it ok to go? [~junrao] Try push some code to open source even harder than play China Shanghai Stock A, hehe. was (Author: waldenchen): Updated reviewboard https://reviews.apache.org/r/33204/diff/ against branch origin/trunk Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: xueqiang wang Labels: newbie, patch Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch, KAFKA-1646_20150616_FixFormat.patch, KAFKA-1646_20150618_235231.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2283) scheduler exception on non-controller node when shutdown
[
https://issues.apache.org/jira/browse/KAFKA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593154#comment-14593154
]
Manikumar Reddy commented on KAFKA-2283:
looks like this got fixed in trunk build.
scheduler exception on non-controller node when shutdown
Key: KAFKA-2283
URL: https://issues.apache.org/jira/browse/KAFKA-2283
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 0.8.2.1
Environment: linux debian
Reporter: allenlee
Assignee: Neha Narkhede
Priority: Minor
When broker shutdown, there is an error log about 'Kafka scheduler has not
been started'.
It only appears on non-controller node. If this broker is the controller, it
shutdown without warning log.
IMHO, *autoRebalanceScheduler.shutdown()* should only valid for controller,
right?
{quote}
[2015-06-17 22:32:51,814] INFO Shutdown complete. (kafka.log.LogManager)
[2015-06-17 22:32:51,815] WARN Kafka scheduler has not been started
(kafka.utils.Utils$)
java.lang.IllegalStateException: Kafka scheduler has not been started
at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
at
kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
at
kafka.controller.KafkaController.shutdown(KafkaController.scala:664)
at
kafka.server.KafkaServer$$anonfun$shutdown$8.apply$mcV$sp(KafkaServer.scala:285)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
at kafka.utils.Utils$.swallowWarn(Utils.scala:45)
at kafka.utils.Logging$class.swallow(Logging.scala:94)
at kafka.utils.Utils$.swallow(Utils.scala:45)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:285)
at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
at kafka.Kafka$$anon$1.run(Kafka.scala:42)
[2015-06-17 22:32:51,818] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593496#comment-14593496 ] Jun Rao commented on KAFKA-2012: The latest patch looks good to me. Could you rebase since we just committed KAFKA-1646? Thanks, Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2249) KafkaConfig does not preserve original Properties
[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593469#comment-14593469 ] Jun Rao commented on KAFKA-2249: Also, I am not sure if LogConfigTest.testFromPropsToProps() is still useful after the patch. KafkaConfig does not preserve original Properties - Key: KAFKA-2249 URL: https://issues.apache.org/jira/browse/KAFKA-2249 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-2249.patch, KAFKA-2249_2015-06-17_17:35:35.patch We typically generate configuration from properties objects (or maps). The old KafkaConfig, and the new ProducerConfig and ConsumerConfig all retain the original Properties object, which means that if the user specified properties that are not part of ConfigDef definitions, they are still accessible. This is important especially for MetricReporters where we want to allow users to pass arbitrary properties for the reporter. One way to support this is by having KafkaConfig implement AbstractConfig, which will give us other nice functionality too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593535#comment-14593535 ] Manikumar Reddy commented on KAFKA-2012: Updated reviewboard https://reviews.apache.org/r/35503/diff/ against branch origin/trunk Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-2012: --- Attachment: KAFKA-2012_2015-06-19_21:09:22.patch Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1451) Broker stuck due to leader election race
[
https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593515#comment-14593515
]
Raghav commented on KAFKA-1451:
---
Hit this issue on version 0.8.2.1 when twiiterstream generate the large data i
have one topic with two broker and two partition
[2015-06-19 20:35:10,141] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-06-19 20:35:16,246] INFO conflict in /brokers/ids/2 data:
{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}
stored data:
{jmx_port:1,timestamp:1434726044184,host:localhost.localdomain,version:1,port:9093}
(kafka.utils.ZkUtils$)
[2015-06-19 20:35:16,796] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-06-19 20:35:22,965] INFO conflict in /brokers/ids/2 data:
{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}
stored data:
{jmx_port:1,timestamp:1434726044184,host:localhost.localdomain,version:1,port:9093}
(kafka.utils.ZkUtils$)
[2015-06-19 20:35:22,967] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-06-19 20:35:29,159] INFO conflict in /brokers/ids/2 data:
{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}
stored data:
{jmx_port:1,timestamp:1434726044184,host:localhost.localdomain,version:1,port:9093}
(kafka.utils.ZkUtils$)
[2015-06-19 20:35:29,161] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-06-19 20:35:35,219] INFO conflict in /brokers/ids/2 data:
{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}
stored data:
{jmx_port:1,timestamp:1434726044184,host:localhost.localdomain,version:1,port:9093}
(kafka.utils.ZkUtils$)
[2015-06-19 20:35:35,221] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-06-19 20:35:41,338] INFO conflict in /brokers/ids/2 data:
{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}
stored data:
{jmx_port:1,timestamp:1434726044184,host:localhost.localdomain,version:1,port:9093}
(kafka.utils.ZkUtils$)
[2015-06-19 20:35:42,208] INFO I wrote this conflicted ephemeral node
[{jmx_port:1,timestamp:1434726183806,host:localhost.localdomain,version:1,port:9093}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
Broker stuck due to leader election race
-
Key: KAFKA-1451
URL: https://issues.apache.org/jira/browse/KAFKA-1451
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Maciek Makowski
Assignee: Manikumar Reddy
Priority: Minor
Labels: newbie
Fix For: 0.8.2.0
Attachments: KAFKA-1451.patch, KAFKA-1451_2014-07-28_20:27:32.patch,
KAFKA-1451_2014-07-29_10:13:23.patch
h3. Symptoms
The broker does not become available due to being stuck in an infinite loop
while electing leader. This can be recognised by the following line being
repeatedly written to server.log:
{code}
[2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:1,timestamp:1400060079108}] at /controller a
while back in a different session, hence I will backoff for this node to be
deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
{code}
h3. Steps to Reproduce
In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely
behave the same with the ZK version included in Kafka distribution) node
setup:
# start both zookeeper and kafka (in any order)
# stop zookeeper
# stop kafka
# start
[jira] [Resolved] (KAFKA-1646) Improve consumer read performance for Windows
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1646. Resolution: Fixed Fix Version/s: 0.8.3 Assignee: Honghai Chen (was: xueqiang wang) Thanks for the patch. Sorry for the delay. +1. Committed to trunk after fixing a compilation error. :core:compileTestScala/Users/junrao/intellij/kafka/core/src/test/scala/unit/kafka/log/LogSegmentTest.scala:233: type mismatch; found : Long required: Int val seg = new LogSegment(tempDir, offset, 10, 1000, 0, SystemTime, fileAlreadyExists = fileAlreadyExists, initFileSize = initFileSize, preallocate = preallocate) ^ one error found FAILED Improve consumer read performance for Windows - Key: KAFKA-1646 URL: https://issues.apache.org/jira/browse/KAFKA-1646 Project: Kafka Issue Type: Improvement Components: log Affects Versions: 0.8.1.1 Environment: Windows Reporter: xueqiang wang Assignee: Honghai Chen Labels: newbie, patch Fix For: 0.8.3 Attachments: Improve consumer read performance for Windows.patch, KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, KAFKA-1646_20150511_AddTestcases.patch, KAFKA-1646_20150609_MergeToLatestTrunk.patch, KAFKA-1646_20150616_FixFormat.patch, KAFKA-1646_20150618_235231.patch This patch is for Window platform only. In Windows platform, if there are more than one replicas writing to disk, the segment log files will not be consistent in disk and then consumer reading performance will be dropped down greatly. This fix allocates more disk spaces when rolling a new segment, and then it will improve the consumer reading performance in NTFS file system. This patch doesn't affect file allocation of other filesystems, for it only adds statements like 'if(Os.iswindow)' or adds methods used on Windows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 35503: Patch for KAFKA-2012
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35503/ --- (Updated June 19, 2015, 3:42 p.m.) Review request for kafka. Bugs: KAFKA-2012 https://issues.apache.org/jira/browse/KAFKA-2012 Repository: kafka Description --- Address Jun's comments Diffs (updated) - core/src/main/scala/kafka/log/Log.scala 6b9274d5f1fff434440e00d9f0e27d32b673076a core/src/test/scala/unit/kafka/log/LogTest.scala a8e57c2348e694a1e0d5407ae8ae105fb1f1bf59 Diff: https://reviews.apache.org/r/35503/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Commented] (KAFKA-2271) transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
[
https://issues.apache.org/jira/browse/KAFKA-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593622#comment-14593622
]
Jason Gustafson commented on KAFKA-2271:
Created reviewboard https://reviews.apache.org/r/35655/diff/
against branch upstream/trunk
transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
Key: KAFKA-2271
URL: https://issues.apache.org/jira/browse/KAFKA-2271
Project: Kafka
Issue Type: Sub-task
Components: core
Reporter: Jun Rao
Attachments: KAFKA-2271.patch
Saw the following transient failure in jenkins.
java.lang.AssertionError: expected:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767, log.cleaner.enable=true,
inter.broker.protocol.version=0.8.3.X, host.name=? ,
log.cleaner.backoff.ms=2080497098, log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414, queued.max.requests=673019914,
controlled.shutdown.max.retries=1810738435, num.replica.fetchers=1160759331,
socket.request.max.bytes=1453815395, log.flush.interval.ms=762170329,
offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606, metrics.sample.window.ms=1000,
offsets.topic.compression.codec=1,
zookeeper.connection.timeout.ms=2068179601,
fetch.purgatory.purge.interval.requests=1242197204,
log.retention.bytes=692466534, log.dirs=/tmp/logs,/tmp/logs2,
replica.fetch.min.bytes=1791426389, compression.type=lz4,
log.roll.jitter.ms=356707666, log.cleaner.threads=2,
replica.lag.time.max.ms=1073834162, advertised.port=4321,
max.connections.per.ip.overrides=127.0.0.1:2, 127.0.0.2:3,
socket.send.buffer.bytes=1319605180, metrics.num.samples=2, port=1234,
replica.fetch.wait.max.ms=321, log.segment.bytes=468671022,
log.retention.minutes=772707425, auto.create.topics.enable=true,
replica.socket.receive.buffer.bytes=1923367476,
log.cleaner.io.max.bytes.per.second=0.2, zookeeper.sync.time.ms=2072589946,
log.roll.jitter.hours=2106718330, log.retention.check.interval.ms=906922522,
reserved.broker.max.id=100, unclean.leader.election.enable=true,
advertised.listeners=PLAINTEXT://:2909,
log.cleaner.io.buffer.load.factor=1.0,
consumer.min.session.timeout.ms=422104288, log.retention.ms=1496447411,
replica.high.watermark.checkpoint.interval.ms=118464842,
log.cleanup.policy=delete, log.cleaner.dedupe.buffer.size=3145729,
offsets.commit.timeout.ms=2084609508, min.insync.replicas=963487957,
zookeeper.connect=127.0.0.1:2181,
leader.imbalance.per.broker.percentage=148038876,
log.index.interval.bytes=242075900,
leader.imbalance.check.interval.seconds=1376263302,
offsets.retention.minutes=1781435041, socket.receive.buffer.bytes=369224522,
log.cleaner.delete.retention.ms=898157008,
replica.socket.timeout.ms=493318414, num.partitions=2,
offsets.topic.segment.bytes=852590082, default.replication.factor=549663639,
log.cleaner.io.buffer.size=905972186, offsets.commit.required.acks=-1,
num.recovery.threads.per.data.dir=1012415473, log.retention.hours=1115262747,
replica.fetch.max.bytes=2041540755, log.roll.hours=115708840,
metric.reporters=, message.max.bytes=1234,
log.cleaner.min.cleanable.ratio=0.6, offsets.load.buffer.size=1818565888,
delete.topic.enable=true, listeners=PLAINTEXT://:9092,
offset.metadata.max.bytes=1563320007,
controlled.shutdown.retry.backoff.ms=1270013702,
max.connections.per.ip=359602609, consumer.max.session.timeout.ms=2124317921,
log.roll.ms=241126032, advertised.host.name=??,
log.flush.scheduler.interval.ms=1548906710,
auto.leader.rebalance.enable=false,
producer.purgatory.purge.interval.requests=1640729755,
controlled.shutdown.enable=false, log.index.size.max.bytes=1748380064,
log.flush.interval.messages=982245822, broker.id=15,
offsets.retention.check.interval.ms=593078788,
replica.fetch.backoff.ms=394858256, background.threads=124969300,
connections.max.idle.ms=554679959} but was:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767,
inter.broker.protocol.version=0.8.3.X, log.cleaner.enable=true,
host.name=?, log.cleaner.backoff.ms=2080497098,
log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414,
controlled.shutdown.max.retries=1810738435, queued.max.requests=673019914,
num.replica.fetchers=1160759331, socket.request.max.bytes=1453815395,
log.flush.interval.ms=762170329, offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
Review Request 35655: Patch for KAFKA-2271
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35655/ --- Review request for kafka. Bugs: KAFKA-2271 https://issues.apache.org/jira/browse/KAFKA-2271 Repository: kafka Description --- KAFKA-2271; fix minor test bugs Diffs - core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 98a5b042a710d3c1064b0379db1d152efc9eabee Diff: https://reviews.apache.org/r/35655/diff/ Testing --- Thanks, Jason Gustafson
[jira] [Updated] (KAFKA-2271) transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
[
https://issues.apache.org/jira/browse/KAFKA-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson updated KAFKA-2271:
---
Attachment: KAFKA-2271.patch
transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
Key: KAFKA-2271
URL: https://issues.apache.org/jira/browse/KAFKA-2271
Project: Kafka
Issue Type: Sub-task
Components: core
Reporter: Jun Rao
Attachments: KAFKA-2271.patch
Saw the following transient failure in jenkins.
java.lang.AssertionError: expected:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767, log.cleaner.enable=true,
inter.broker.protocol.version=0.8.3.X, host.name=? ,
log.cleaner.backoff.ms=2080497098, log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414, queued.max.requests=673019914,
controlled.shutdown.max.retries=1810738435, num.replica.fetchers=1160759331,
socket.request.max.bytes=1453815395, log.flush.interval.ms=762170329,
offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606, metrics.sample.window.ms=1000,
offsets.topic.compression.codec=1,
zookeeper.connection.timeout.ms=2068179601,
fetch.purgatory.purge.interval.requests=1242197204,
log.retention.bytes=692466534, log.dirs=/tmp/logs,/tmp/logs2,
replica.fetch.min.bytes=1791426389, compression.type=lz4,
log.roll.jitter.ms=356707666, log.cleaner.threads=2,
replica.lag.time.max.ms=1073834162, advertised.port=4321,
max.connections.per.ip.overrides=127.0.0.1:2, 127.0.0.2:3,
socket.send.buffer.bytes=1319605180, metrics.num.samples=2, port=1234,
replica.fetch.wait.max.ms=321, log.segment.bytes=468671022,
log.retention.minutes=772707425, auto.create.topics.enable=true,
replica.socket.receive.buffer.bytes=1923367476,
log.cleaner.io.max.bytes.per.second=0.2, zookeeper.sync.time.ms=2072589946,
log.roll.jitter.hours=2106718330, log.retention.check.interval.ms=906922522,
reserved.broker.max.id=100, unclean.leader.election.enable=true,
advertised.listeners=PLAINTEXT://:2909,
log.cleaner.io.buffer.load.factor=1.0,
consumer.min.session.timeout.ms=422104288, log.retention.ms=1496447411,
replica.high.watermark.checkpoint.interval.ms=118464842,
log.cleanup.policy=delete, log.cleaner.dedupe.buffer.size=3145729,
offsets.commit.timeout.ms=2084609508, min.insync.replicas=963487957,
zookeeper.connect=127.0.0.1:2181,
leader.imbalance.per.broker.percentage=148038876,
log.index.interval.bytes=242075900,
leader.imbalance.check.interval.seconds=1376263302,
offsets.retention.minutes=1781435041, socket.receive.buffer.bytes=369224522,
log.cleaner.delete.retention.ms=898157008,
replica.socket.timeout.ms=493318414, num.partitions=2,
offsets.topic.segment.bytes=852590082, default.replication.factor=549663639,
log.cleaner.io.buffer.size=905972186, offsets.commit.required.acks=-1,
num.recovery.threads.per.data.dir=1012415473, log.retention.hours=1115262747,
replica.fetch.max.bytes=2041540755, log.roll.hours=115708840,
metric.reporters=, message.max.bytes=1234,
log.cleaner.min.cleanable.ratio=0.6, offsets.load.buffer.size=1818565888,
delete.topic.enable=true, listeners=PLAINTEXT://:9092,
offset.metadata.max.bytes=1563320007,
controlled.shutdown.retry.backoff.ms=1270013702,
max.connections.per.ip=359602609, consumer.max.session.timeout.ms=2124317921,
log.roll.ms=241126032, advertised.host.name=??,
log.flush.scheduler.interval.ms=1548906710,
auto.leader.rebalance.enable=false,
producer.purgatory.purge.interval.requests=1640729755,
controlled.shutdown.enable=false, log.index.size.max.bytes=1748380064,
log.flush.interval.messages=982245822, broker.id=15,
offsets.retention.check.interval.ms=593078788,
replica.fetch.backoff.ms=394858256, background.threads=124969300,
connections.max.idle.ms=554679959} but was:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767,
inter.broker.protocol.version=0.8.3.X, log.cleaner.enable=true,
host.name=?, log.cleaner.backoff.ms=2080497098,
log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414,
controlled.shutdown.max.retries=1810738435, queued.max.requests=673019914,
num.replica.fetchers=1160759331, socket.request.max.bytes=1453815395,
log.flush.interval.ms=762170329, offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606, metrics.sample.window.ms=1000,
offsets.topic.compression.codec=1,
[jira] [Updated] (KAFKA-2271) transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
[
https://issues.apache.org/jira/browse/KAFKA-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson updated KAFKA-2271:
---
Assignee: Jason Gustafson
Status: Patch Available (was: Open)
transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
Key: KAFKA-2271
URL: https://issues.apache.org/jira/browse/KAFKA-2271
Project: Kafka
Issue Type: Sub-task
Components: core
Reporter: Jun Rao
Assignee: Jason Gustafson
Attachments: KAFKA-2271.patch
Saw the following transient failure in jenkins.
java.lang.AssertionError: expected:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767, log.cleaner.enable=true,
inter.broker.protocol.version=0.8.3.X, host.name=? ,
log.cleaner.backoff.ms=2080497098, log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414, queued.max.requests=673019914,
controlled.shutdown.max.retries=1810738435, num.replica.fetchers=1160759331,
socket.request.max.bytes=1453815395, log.flush.interval.ms=762170329,
offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606, metrics.sample.window.ms=1000,
offsets.topic.compression.codec=1,
zookeeper.connection.timeout.ms=2068179601,
fetch.purgatory.purge.interval.requests=1242197204,
log.retention.bytes=692466534, log.dirs=/tmp/logs,/tmp/logs2,
replica.fetch.min.bytes=1791426389, compression.type=lz4,
log.roll.jitter.ms=356707666, log.cleaner.threads=2,
replica.lag.time.max.ms=1073834162, advertised.port=4321,
max.connections.per.ip.overrides=127.0.0.1:2, 127.0.0.2:3,
socket.send.buffer.bytes=1319605180, metrics.num.samples=2, port=1234,
replica.fetch.wait.max.ms=321, log.segment.bytes=468671022,
log.retention.minutes=772707425, auto.create.topics.enable=true,
replica.socket.receive.buffer.bytes=1923367476,
log.cleaner.io.max.bytes.per.second=0.2, zookeeper.sync.time.ms=2072589946,
log.roll.jitter.hours=2106718330, log.retention.check.interval.ms=906922522,
reserved.broker.max.id=100, unclean.leader.election.enable=true,
advertised.listeners=PLAINTEXT://:2909,
log.cleaner.io.buffer.load.factor=1.0,
consumer.min.session.timeout.ms=422104288, log.retention.ms=1496447411,
replica.high.watermark.checkpoint.interval.ms=118464842,
log.cleanup.policy=delete, log.cleaner.dedupe.buffer.size=3145729,
offsets.commit.timeout.ms=2084609508, min.insync.replicas=963487957,
zookeeper.connect=127.0.0.1:2181,
leader.imbalance.per.broker.percentage=148038876,
log.index.interval.bytes=242075900,
leader.imbalance.check.interval.seconds=1376263302,
offsets.retention.minutes=1781435041, socket.receive.buffer.bytes=369224522,
log.cleaner.delete.retention.ms=898157008,
replica.socket.timeout.ms=493318414, num.partitions=2,
offsets.topic.segment.bytes=852590082, default.replication.factor=549663639,
log.cleaner.io.buffer.size=905972186, offsets.commit.required.acks=-1,
num.recovery.threads.per.data.dir=1012415473, log.retention.hours=1115262747,
replica.fetch.max.bytes=2041540755, log.roll.hours=115708840,
metric.reporters=, message.max.bytes=1234,
log.cleaner.min.cleanable.ratio=0.6, offsets.load.buffer.size=1818565888,
delete.topic.enable=true, listeners=PLAINTEXT://:9092,
offset.metadata.max.bytes=1563320007,
controlled.shutdown.retry.backoff.ms=1270013702,
max.connections.per.ip=359602609, consumer.max.session.timeout.ms=2124317921,
log.roll.ms=241126032, advertised.host.name=??,
log.flush.scheduler.interval.ms=1548906710,
auto.leader.rebalance.enable=false,
producer.purgatory.purge.interval.requests=1640729755,
controlled.shutdown.enable=false, log.index.size.max.bytes=1748380064,
log.flush.interval.messages=982245822, broker.id=15,
offsets.retention.check.interval.ms=593078788,
replica.fetch.backoff.ms=394858256, background.threads=124969300,
connections.max.idle.ms=554679959} but was:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767,
inter.broker.protocol.version=0.8.3.X, log.cleaner.enable=true,
host.name=?, log.cleaner.backoff.ms=2080497098,
log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414,
controlled.shutdown.max.retries=1810738435, queued.max.requests=673019914,
num.replica.fetchers=1160759331, socket.request.max.bytes=1453815395,
log.flush.interval.ms=762170329, offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606,
[jira] [Commented] (KAFKA-2249) KafkaConfig does not preserve original Properties
[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593555#comment-14593555 ] Gwen Shapira commented on KAFKA-2249: - Agree that the test is kinda silly now :) And yes, the printouts were useful when debugging test cases with 1 topic and 1 partition, but PITA in production. OTOH, printout of non-default config per topic in a debug level INFO or DEBUG makes sense IMO (even on large systems) - it can be super useful when people send logs and ask for help. I'll see if we can do it per topic without making too much of a mess. Mind if I take those fixes in separate JIRA? KafkaConfig does not preserve original Properties - Key: KAFKA-2249 URL: https://issues.apache.org/jira/browse/KAFKA-2249 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-2249.patch, KAFKA-2249_2015-06-17_17:35:35.patch We typically generate configuration from properties objects (or maps). The old KafkaConfig, and the new ProducerConfig and ConsumerConfig all retain the original Properties object, which means that if the user specified properties that are not part of ConfigDef definitions, they are still accessible. This is important especially for MetricReporters where we want to allow users to pass arbitrary properties for the reporter. One way to support this is by having KafkaConfig implement AbstractConfig, which will give us other nice functionality too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2271) transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
[
https://issues.apache.org/jira/browse/KAFKA-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593646#comment-14593646
]
Jason Gustafson commented on KAFKA-2271:
This patch may or may not fix the problem, but the only apparent difference in
the output above is the extra space after the random host.name field. Since the
randomness of that field (and the advertised host name) didn't seem significant
to the test, I just used a static string instead.
I also noticed a couple other minor problems which could cause the test to fail.
transient unit test failure in KafkaConfigConfigDefTest.testFromPropsToProps
Key: KAFKA-2271
URL: https://issues.apache.org/jira/browse/KAFKA-2271
Project: Kafka
Issue Type: Sub-task
Components: core
Reporter: Jun Rao
Assignee: Jason Gustafson
Attachments: KAFKA-2271.patch
Saw the following transient failure in jenkins.
java.lang.AssertionError: expected:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767, log.cleaner.enable=true,
inter.broker.protocol.version=0.8.3.X, host.name=? ,
log.cleaner.backoff.ms=2080497098, log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414, queued.max.requests=673019914,
controlled.shutdown.max.retries=1810738435, num.replica.fetchers=1160759331,
socket.request.max.bytes=1453815395, log.flush.interval.ms=762170329,
offsets.topic.replication.factor=1011,
log.flush.offset.checkpoint.interval.ms=923125288,
security.inter.broker.protocol=PLAINTEXT,
zookeeper.session.timeout.ms=413974606, metrics.sample.window.ms=1000,
offsets.topic.compression.codec=1,
zookeeper.connection.timeout.ms=2068179601,
fetch.purgatory.purge.interval.requests=1242197204,
log.retention.bytes=692466534, log.dirs=/tmp/logs,/tmp/logs2,
replica.fetch.min.bytes=1791426389, compression.type=lz4,
log.roll.jitter.ms=356707666, log.cleaner.threads=2,
replica.lag.time.max.ms=1073834162, advertised.port=4321,
max.connections.per.ip.overrides=127.0.0.1:2, 127.0.0.2:3,
socket.send.buffer.bytes=1319605180, metrics.num.samples=2, port=1234,
replica.fetch.wait.max.ms=321, log.segment.bytes=468671022,
log.retention.minutes=772707425, auto.create.topics.enable=true,
replica.socket.receive.buffer.bytes=1923367476,
log.cleaner.io.max.bytes.per.second=0.2, zookeeper.sync.time.ms=2072589946,
log.roll.jitter.hours=2106718330, log.retention.check.interval.ms=906922522,
reserved.broker.max.id=100, unclean.leader.election.enable=true,
advertised.listeners=PLAINTEXT://:2909,
log.cleaner.io.buffer.load.factor=1.0,
consumer.min.session.timeout.ms=422104288, log.retention.ms=1496447411,
replica.high.watermark.checkpoint.interval.ms=118464842,
log.cleanup.policy=delete, log.cleaner.dedupe.buffer.size=3145729,
offsets.commit.timeout.ms=2084609508, min.insync.replicas=963487957,
zookeeper.connect=127.0.0.1:2181,
leader.imbalance.per.broker.percentage=148038876,
log.index.interval.bytes=242075900,
leader.imbalance.check.interval.seconds=1376263302,
offsets.retention.minutes=1781435041, socket.receive.buffer.bytes=369224522,
log.cleaner.delete.retention.ms=898157008,
replica.socket.timeout.ms=493318414, num.partitions=2,
offsets.topic.segment.bytes=852590082, default.replication.factor=549663639,
log.cleaner.io.buffer.size=905972186, offsets.commit.required.acks=-1,
num.recovery.threads.per.data.dir=1012415473, log.retention.hours=1115262747,
replica.fetch.max.bytes=2041540755, log.roll.hours=115708840,
metric.reporters=, message.max.bytes=1234,
log.cleaner.min.cleanable.ratio=0.6, offsets.load.buffer.size=1818565888,
delete.topic.enable=true, listeners=PLAINTEXT://:9092,
offset.metadata.max.bytes=1563320007,
controlled.shutdown.retry.backoff.ms=1270013702,
max.connections.per.ip=359602609, consumer.max.session.timeout.ms=2124317921,
log.roll.ms=241126032, advertised.host.name=??,
log.flush.scheduler.interval.ms=1548906710,
auto.leader.rebalance.enable=false,
producer.purgatory.purge.interval.requests=1640729755,
controlled.shutdown.enable=false, log.index.size.max.bytes=1748380064,
log.flush.interval.messages=982245822, broker.id=15,
offsets.retention.check.interval.ms=593078788,
replica.fetch.backoff.ms=394858256, background.threads=124969300,
connections.max.idle.ms=554679959} but was:{num.io.threads=2051678117,
log.dir=/tmp/log, num.network.threads=442579598,
offsets.topic.num.partitions=1996793767,
inter.broker.protocol.version=0.8.3.X, log.cleaner.enable=true,
host.name=?, log.cleaner.backoff.ms=2080497098,
log.segment.delete.delay.ms=516834257,
controller.socket.timeout.ms=11414,
Re: Review Request 34789: Patch for KAFKA-2168
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/ --- (Updated June 19, 2015, 4:19 p.m.) Review request for kafka. Bugs: KAFKA-2168 https://issues.apache.org/jira/browse/KAFKA-2168 Repository: kafka Description (updated) --- KAFKA-2168; refactored callback handling to prevent unnecessary requests KAFKA-2168; address review comments KAFKA-2168; fix rebase error and checkstyle issue KAFKA-2168; address review comments and add docs KAFKA-2168; handle polling with timeout 0 KAFKA-2168; timeout=0 means return immediately KAFKA-2168; address review comments KAFKA-2168; address more review comments KAFKA-2168; updated for review comments Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 8f587bc0705b65b3ef37c86e0c25bb43ab8803de clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java 1ca75f83d3667f7d01da1ae2fd9488fb79562364 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerWakeupException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 951c34c92710fc4b38d656e99d2a41255c60aeb7 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java f50da825756938c193d7f07bee953e000e2627d9 clients/src/main/java/org/apache/kafka/clients/consumer/OffsetResetStrategy.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java 41cb9458f51875ac9418fce52f264b35adba92f4 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 56281ee15cc33dfc96ff64d5b1e596047c7132a4 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Heartbeat.java e7cfaaad296fa6e325026a5eee1aaf9b9c0fe1fe clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestFuture.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java cee75410127dd1b86c1156563003216d93a086b3 clients/src/main/java/org/apache/kafka/common/utils/Utils.java f73eedb030987f018d8446bb1dcd98d19fa97331 clients/src/test/java/org/apache/kafka/clients/consumer/MockConsumerTest.java 677edd385f35d4262342b567262c0b874876d25b clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java 1454ab73df22cce028f41f74b970628829da4e9d clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java 419541011d652becf0cda7a5e62ce813cddb1732 clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatTest.java ecc78cedf59a994fcf084fa7a458fe9ed5386b00 clients/src/test/java/org/apache/kafka/clients/consumer/internals/SubscriptionStateTest.java e000cf8e10ebfacd6c9ee68d7b88ff8c157f73c6 clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 2ebe3c21f611dc133a2dbb8c7dfb0845f8c21498 Diff: https://reviews.apache.org/r/34789/diff/ Testing --- Thanks, Jason Gustafson
[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593579#comment-14593579 ] Jason Gustafson commented on KAFKA-2168: Updated reviewboard https://reviews.apache.org/r/34789/diff/ against branch upstream/trunk New consumer poll() can block other calls like position(), commit(), and close() indefinitely - Key: KAFKA-2168 URL: https://issues.apache.org/jira/browse/KAFKA-2168 Project: Kafka Issue Type: Sub-task Components: clients, consumer Reporter: Ewen Cheslack-Postava Assignee: Jason Gustafson Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-2168.patch, KAFKA-2168_2015-06-01_16:03:38.patch, KAFKA-2168_2015-06-02_17:09:37.patch, KAFKA-2168_2015-06-03_18:20:23.patch, KAFKA-2168_2015-06-03_21:06:42.patch, KAFKA-2168_2015-06-04_14:36:04.patch, KAFKA-2168_2015-06-05_12:01:28.patch, KAFKA-2168_2015-06-05_12:44:48.patch, KAFKA-2168_2015-06-11_14:09:59.patch, KAFKA-2168_2015-06-18_14:39:36.patch, KAFKA-2168_2015-06-19_09:19:02.patch The new consumer is currently using very coarse-grained synchronization. For most methods this isn't a problem since they finish quickly once the lock is acquired, but poll() might run for a long time (and commonly will since polling with long timeouts is a normal use case). This means any operations invoked from another thread may block until the poll() call completes. Some example use cases where this can be a problem: * A shutdown hook is registered to trigger shutdown and invokes close(). It gets invoked from another thread and blocks indefinitely. * User wants to manage offset commit themselves in a background thread. If the commit policy is not purely time based, it's not currently possibly to make sure the call to commit() will be processed promptly. Two possible solutions to this: 1. Make sure a lock is not held during the actual select call. Since we have multiple layers (KafkaConsumer - NetworkClient - Selector - nio Selector) this is probably hard to make work cleanly since locking is currently only performed at the KafkaConsumer level and we'd want it unlocked around a single line of code in Selector. 2. Wake up the selector before synchronizing for certain operations. This would require some additional coordination to make sure the caller of wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() thread being woken up and then promptly reacquiring the lock with a subsequent long poll() call). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2168: --- Attachment: KAFKA-2168_2015-06-19_09:19:02.patch New consumer poll() can block other calls like position(), commit(), and close() indefinitely - Key: KAFKA-2168 URL: https://issues.apache.org/jira/browse/KAFKA-2168 Project: Kafka Issue Type: Sub-task Components: clients, consumer Reporter: Ewen Cheslack-Postava Assignee: Jason Gustafson Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-2168.patch, KAFKA-2168_2015-06-01_16:03:38.patch, KAFKA-2168_2015-06-02_17:09:37.patch, KAFKA-2168_2015-06-03_18:20:23.patch, KAFKA-2168_2015-06-03_21:06:42.patch, KAFKA-2168_2015-06-04_14:36:04.patch, KAFKA-2168_2015-06-05_12:01:28.patch, KAFKA-2168_2015-06-05_12:44:48.patch, KAFKA-2168_2015-06-11_14:09:59.patch, KAFKA-2168_2015-06-18_14:39:36.patch, KAFKA-2168_2015-06-19_09:19:02.patch The new consumer is currently using very coarse-grained synchronization. For most methods this isn't a problem since they finish quickly once the lock is acquired, but poll() might run for a long time (and commonly will since polling with long timeouts is a normal use case). This means any operations invoked from another thread may block until the poll() call completes. Some example use cases where this can be a problem: * A shutdown hook is registered to trigger shutdown and invokes close(). It gets invoked from another thread and blocks indefinitely. * User wants to manage offset commit themselves in a background thread. If the commit policy is not purely time based, it's not currently possibly to make sure the call to commit() will be processed promptly. Two possible solutions to this: 1. Make sure a lock is not held during the actual select call. Since we have multiple layers (KafkaConsumer - NetworkClient - Selector - nio Selector) this is probably hard to make work cleanly since locking is currently only performed at the KafkaConsumer level and we'd want it unlocked around a single line of code in Selector. 2. Wake up the selector before synchronizing for certain operations. This would require some additional coordination to make sure the caller of wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() thread being woken up and then promptly reacquiring the lock with a subsequent long poll() call). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2012) Broker should automatically handle corrupt index files
[ https://issues.apache.org/jira/browse/KAFKA-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593812#comment-14593812 ] Mayuresh Gharat edited comment on KAFKA-2012 at 6/19/15 7:30 PM: - Discussed this with [~jjkoshy]. This patch seems like a workaround and does not actually tell us why the file got corrupted in first place. We can probably have a config that can turn this code path ON or OFF, so that we can actually investigate when this happens. Let me know, I can open another ticket or use this : https://issues.apache.org/jira/browse/KAFKA-1554 to add that config. This was discussed in KAFKA-1554 : Joel Koshy added a comment - 14/Mar/15 01:10 That would be a work-around, but ideally we should figure out why it happened in the first place. Jun Rao added a comment - 09/Apr/15 02:06 Yes, I am not sure if auto fixing the index is better. People then may not realize if there is an issue. It would be better to figure out what's causing this. Thanks, Mayuresh was (Author: mgharat): Discussed this with [~jjkoshy]. This patch seems like a workaround and does not actually tell us why the file got corrupted in first place. We can probably have a config that can turn this code path ON or OFF, so that we can actually investigate when this happens. Let me know, I can open another ticket or use this : https://issues.apache.org/jira/browse/KAFKA-1554 to add that config. Thanks, Mayuresh Broker should automatically handle corrupt index files -- Key: KAFKA-2012 URL: https://issues.apache.org/jira/browse/KAFKA-2012 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1 Reporter: Todd Palino Assignee: Manikumar Reddy Fix For: 0.8.3 Attachments: KAFKA-2012.patch, KAFKA-2012_2015-06-19_18:55:11.patch, KAFKA-2012_2015-06-19_21:09:22.patch We had a bunch of unclean system shutdowns (power failure), which caused corruption on our disks holding log segments in many cases. While the broker is handling the log segment corruption properly (truncation), it is having problems with corruption in the index files. Additionally, this only seems to be happening on some startups (while we are upgrading). The broker should just do what I do when I hit a corrupt index file - remove it and rebuild it. 2015/03/09 17:58:53.873 FATAL [KafkaServerStartable] [main] [kafka-server] [] Fatal error during KafkaServerStartable startup. Prepare to shutdown java.lang.IllegalArgumentException: requirement failed: Corrupt index found, index file (/export/content/kafka/i001_caches/__consumer_offsets-39/.index) has non-zero size but the last offset is -2121629628 and the base offset is 0 at scala.Predef$.require(Predef.scala:233) at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:185) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.Log.loadSegments(Log.scala:184) at kafka.log.Log.init(Log.scala:82) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:141) at kafka.utils.Utils$$anon$1.run(Utils.scala:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2287) Add metric to track the number of clientId throttled
Aditya Auradkar created KAFKA-2287: -- Summary: Add metric to track the number of clientId throttled Key: KAFKA-2287 URL: https://issues.apache.org/jira/browse/KAFKA-2287 Project: Kafka Issue Type: Sub-task Reporter: Aditya Auradkar Assignee: Aditya Auradkar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
