Build failed in Jenkins: Kafka-trunk #395
See https://builds.apache.org/job/Kafka-trunk/395/changes Changes: [jjkoshy] KAFKA-1959; Rename group to groupId in TestOffsetManager due to collision with Thread.group in IBM's JDK; reviewed by Joel Koshy and Gwen Shapira [jjkoshy] KAFKA-1960; .gitignore does not exclude test generated files and folders; reviewed by Joel Koshy and Gwen Shapira -- [...truncated 558 lines...] kafka.admin.AddPartitionsTest testManualAssignmentOfReplicas PASSED kafka.admin.AddPartitionsTest testReplicaPlacement PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithAllAliveReplicas PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicWithRecoveredFollower PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicOnControllerFailover PASSED kafka.admin.DeleteTopicTest testPartitionReassignmentDuringDeleteTopic PASSED kafka.admin.DeleteTopicTest testDeleteTopicDuringAddPartition PASSED kafka.admin.DeleteTopicTest testAddPartitionDuringDeleteTopic PASSED kafka.admin.DeleteTopicTest testRecreateTopicAfterDeletion PASSED kafka.admin.DeleteTopicTest testDeleteNonExistingTopic PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithCleaner PASSED kafka.api.ConsumerTest testSimpleConsumption PASSED kafka.api.ConsumerTest testAutoOffsetReset PASSED kafka.api.ConsumerTest testSeek PASSED kafka.api.ConsumerTest testGroupConsumption PASSED kafka.api.ConsumerTest testPositionAndCommit PASSED kafka.api.ConsumerTest testPartitionsFor PASSED kafka.api.ConsumerTest testConsumptionWithBrokerFailures PASSED kafka.api.ConsumerTest testSeekAndCommitWithBrokerFailures PASSED kafka.api.ConsumerTest testPartitionReassignmentCallback PASSED kafka.api.RequestResponseSerializationTest testSerializationAndDeserialization PASSED kafka.api.ApiUtilsTest testShortStringNonASCII PASSED kafka.api.ApiUtilsTest testShortStringASCII PASSED kafka.api.test.ProducerSendTest testSendOffset PASSED kafka.api.test.ProducerSendTest testSerializer PASSED kafka.api.test.ProducerSendTest testClose PASSED kafka.api.test.ProducerSendTest testSendToPartition PASSED kafka.api.test.ProducerSendTest testAutoCreateTopic PASSED kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicas PASSED kafka.api.test.ProducerFailureHandlingTest testInvalidPartition PASSED kafka.api.test.ProducerFailureHandlingTest testTooLargeRecordWithAckZero PASSED kafka.api.test.ProducerFailureHandlingTest testTooLargeRecordWithAckOne PASSED kafka.api.test.ProducerFailureHandlingTest testNonExistentTopic PASSED kafka.api.test.ProducerFailureHandlingTest testWrongBrokerList PASSED kafka.api.test.ProducerFailureHandlingTest testNoResponse PASSED kafka.api.test.ProducerFailureHandlingTest testSendAfterClosed PASSED kafka.api.test.ProducerFailureHandlingTest testBrokerFailure PASSED kafka.api.test.ProducerFailureHandlingTest testCannotSendToInternalTopic PASSED kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicasAfterBrokerShutdown PASSED kafka.api.test.ProducerCompressionTest testCompression[0] PASSED kafka.api.test.ProducerCompressionTest testCompression[1] PASSED kafka.api.test.ProducerCompressionTest testCompression[2] PASSED kafka.api.test.ProducerCompressionTest testCompression[3] PASSED kafka.javaapi.consumer.ZookeeperConsumerConnectorTest testBasic PASSED kafka.javaapi.message.ByteBufferMessageSetTest testWrittenEqualsRead PASSED kafka.javaapi.message.ByteBufferMessageSetTest testIteratorIsConsistent PASSED kafka.javaapi.message.ByteBufferMessageSetTest testSizeInBytes PASSED kafka.javaapi.message.ByteBufferMessageSetTest testIteratorIsConsistentWithCompression PASSED kafka.javaapi.message.ByteBufferMessageSetTest testSizeInBytesWithCompression PASSED kafka.javaapi.message.ByteBufferMessageSetTest testEquals PASSED kafka.javaapi.message.ByteBufferMessageSetTest testEqualsWithCompression PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionEnabled PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionDisabled PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionEnabledByTopicOverride PASSED kafka.integration.UncleanLeaderElectionTest testCleanLeaderElectionDisabledByTopicOverride PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionInvalidTopicOverride PASSED kafka.integration.FetcherTest testFetcher PASSED kafka.integration.RollingBounceTest testRollingBounce PASSED kafka.integration.PrimitiveApiTest testFetchRequestCanProperlySerialize PASSED kafka.integration.PrimitiveApiTest testEmptyFetchRequest PASSED kafka.integration.PrimitiveApiTest testDefaultEncoderProducerAndFetch PASSED kafka.integration.PrimitiveApiTest testDefaultEncoderProducerAndFetchWithCompression PASSED kafka.integration.PrimitiveApiTest testProduceAndMultiFetch PASSED kafka.integration.PrimitiveApiTest testMultiProduce PASSED
Re: [jira] [Resolved] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors
Joel, thanks so much for merging the two sets.
Tong
Sent from my iPhone
On Feb 18, 2015, at 2:55 AM, Joel Koshy (JIRA) j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Koshy resolved KAFKA-1959.
---
Resolution: Fixed
Assignee: Tong Li
Thanks for the patch - committed to trunk.
Class CommitThread overwrite group of Thread class causing compile
errors
-
Key: KAFKA-1959
URL: https://issues.apache.org/jira/browse/KAFKA-1959
Project: Kafka
Issue Type: Bug
Components: core
Environment: scala 2.10.4
Reporter: Tong Li
Assignee: Tong Li
Labels: newbie
Attachments: KAFKA-1959.patch, compileError.png
class CommitThread(id: Int, partitionCount: Int, commitIntervalMs:
Long, zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {
private val group = group- + id
group overwrite class Thread group member, causing the following
compile error:
overriding variable group in class Thread of type ThreadGroup; value
group has weaker access privileges; it should not be private
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Build failed in Jenkins: KafkaPreCommit #8
See https://builds.apache.org/job/KafkaPreCommit/8/changes Changes: [jjkoshy] KAFKA-1959; Rename group to groupId in TestOffsetManager due to collision with Thread.group in IBM's JDK; reviewed by Joel Koshy and Gwen Shapira [jjkoshy] KAFKA-1960; .gitignore does not exclude test generated files and folders; reviewed by Joel Koshy and Gwen Shapira -- [...truncated 551 lines...] kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.consumer.ConsumerIteratorTest testConsumerIteratorDeduplicationDeepIterator PASSED kafka.consumer.ConsumerIteratorTest testConsumerIteratorDecodingFailure PASSED kafka.consumer.ZookeeperConsumerConnectorTest testCompression PASSED kafka.consumer.ZookeeperConsumerConnectorTest testBasic PASSED kafka.consumer.ZookeeperConsumerConnectorTest testCompressionSetConsumption PASSED kafka.consumer.ZookeeperConsumerConnectorTest testConsumerDecoder PASSED kafka.consumer.ZookeeperConsumerConnectorTest testLeaderSelectionForPartition PASSED kafka.consumer.ZookeeperConsumerConnectorTest testConsumerRebalanceListener PASSED kafka.consumer.TopicFilterTest testWhitelists PASSED kafka.consumer.TopicFilterTest testBlacklists PASSED kafka.consumer.TopicFilterTest testWildcardTopicCountGetTopicCountMapEscapeJson PASSED kafka.consumer.MetricsTest testMetricsLeak PASSED kafka.zk.ZKEphemeralTest testEphemeralNodeCleanup PASSED kafka.server.ReplicaManagerTest testHighWaterMarkDirectoryMapping PASSED kafka.server.ReplicaManagerTest testHighwaterMarkRelativeDirectoryMapping PASSED kafka.server.ReplicaManagerTest testIllegalRequiredAcks PASSED kafka.server.IsrExpirationTest testIsrExpirationForStuckFollowers PASSED kafka.server.IsrExpirationTest testIsrExpirationForSlowFollowers PASSED kafka.server.SimpleFetchTest testReadFromLog FAILED junit.framework.AssertionFailedError: Counts should increment after fetch expected:3 but was:14759 at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:277) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:130) at kafka.server.SimpleFetchTest.testReadFromLog(SimpleFetchTest.scala:145) kafka.server.ServerGenerateBrokerIdTest testAutoGenerateBrokerId PASSED kafka.server.ServerGenerateBrokerIdTest testUserConfigAndGeneratedBrokerId PASSED kafka.server.ServerGenerateBrokerIdTest testMultipleLogDirsMetaProps PASSED kafka.server.ServerGenerateBrokerIdTest testConsistentBrokerIdFromUserConfigAndMetaProps PASSED kafka.server.ServerShutdownTest testCleanShutdown PASSED kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled PASSED kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup PASSED kafka.server.ServerShutdownTest testConsecutiveShutdown PASSED kafka.server.KafkaConfigTest testLogRetentionTimeHoursProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeMinutesProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeMsProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeNoConfigProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeBothMinutesAndHoursProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeBothMinutesAndMsProvided PASSED kafka.server.KafkaConfigTest testAdvertiseDefaults PASSED kafka.server.KafkaConfigTest testAdvertiseConfigured PASSED kafka.server.KafkaConfigTest testUncleanLeaderElectionDefault PASSED kafka.server.KafkaConfigTest testUncleanElectionDisabled PASSED kafka.server.KafkaConfigTest testUncleanElectionEnabled PASSED kafka.server.KafkaConfigTest testUncleanElectionInvalid PASSED kafka.server.KafkaConfigTest testLogRollTimeMsProvided PASSED kafka.server.KafkaConfigTest testLogRollTimeBothMsAndHoursProvided PASSED kafka.server.KafkaConfigTest testLogRollTimeNoConfigProvided PASSED kafka.server.KafkaConfigTest testDefaultCompressionType PASSED kafka.server.KafkaConfigTest testValidCompressionType PASSED kafka.server.KafkaConfigTest testInvalidCompressionType PASSED kafka.server.OffsetCommitTest testUpdateOffsets PASSED kafka.server.OffsetCommitTest testCommitAndFetchOffsets PASSED kafka.server.OffsetCommitTest testLargeMetadataPayload PASSED kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeLatestTime PASSED kafka.server.LogOffsetTest testEmptyLogsGetOffsets PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeNow PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeEarliestTime PASSED kafka.server.AdvertiseBrokerTest testBrokerAdvertiseToZK PASSED kafka.server.ServerStartupTest testBrokerCreatesZKChroot PASSED kafka.server.DelayedOperationTest testRequestSatisfaction PASSED kafka.server.DelayedOperationTest testRequestExpiry PASSED
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Hey Andrii,
Generally we can do good error handling without needing custom server-side
messages. I.e. generally the client has the context to know that if it got
an error that the topic doesn't exist to say Topic X doesn't exist rather
than error code 14 (or whatever). Maybe there are specific cases where
this is hard? If we want to add server-side error messages we really do
need to do this in a consistent way across the protocol.
I still have a bunch of open questions here from my previous list. I will
be out for the next few days for Strata though. Maybe we could do a Google
Hangout chat on any open issues some time towards the end of next week for
anyone interested in this ticket? I have a feeling that might progress
things a little faster than email--I think we could talk through those
issues I brought up fairly quickly...
-Jay
On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
Hi all,
I'm trying to address some of the issues which were mentioned earlier about
Admin RQ/RP format. One of those was about batching operations. What if we
follow TopicCommand approach and let people specify topic-name by regexp -
would that cover most of the use cases?
Secondly, is what information should we generally provide in Admin
responses.
I realize that Admin commands don't imply they will be used only in CLI
but,
it seems to me, CLI is a very important client of this feature. In this
case,
seems logical, we would like to provide users with rich experience in terms
of
getting results / errors of the executed commands. Usually we supply with
responses only errorCode, which looks very limiting, in case of CLI we may
want to print human readable error description.
So, taking into account previous item about batching, what do you think
about
having smth like:
('create' doesn't support regexp)
CreateTopicRequest = TopicName Partitions Replicas ReplicaAssignment
[Config]
CreateTopicResponse = ErrorCode ErrorDescription
ErrorCode = int16
ErrorDescription = string (empty if successful)
AlterTopicRequest - TopicNameRegexp Partitions ReplicaAssignment
[AddedConfig] [DeletedConfig]
AlterTopicResponse - [TopicName ErrorCode ErrorDescription]
CommandErrorCode CommandErrorDescription
CommandErrorCode = int16
CommandErrorDescription = string (nonempty in case of fatal error, e.g.
we couldn't get topics by regexp)
DescribeTopicRequest - TopicNameRegexp
DescribeTopicResponse - [TopicName TopicDescription ErrorCode
ErrorDescription] CommandErrorCode CommandErrorDescription
Also, any thoughts about our discussion regarding re-routing facility? In
my
understanding, it is like between augmenting TopicMetadataRequest
(to include at least controllerId) and implementing new generic re-routing
facility so sending messages to controller will be handled by it.
Thanks,
Andrii Biletskyi
On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
@Guozhang:
Thanks for your comments, I've answered some of those. The main thing is
having merged request for create-alter-delete-describe - I have some
concerns about this approach.
@*Jay*:
I see that introduced ClusterMetadaRequest is also one of the concerns.
We
can solve it if we implement re-routing facility. But I agree with
Guozhang - it will make clients' internals a little bit easier but this
seems to be a complex logic to implement and support then. Especially for
Fetch and Produce (even if we add re-routing later for these requests).
Also people will tend to avoid this re-routing facility and hold local
cluster cache to ensure their high-priority requests (which some of the
admin requests are) not sent to some busy broker where they wait to be
routed to the correct one.
As pointed out by Jun here (
https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530
)
to solve the issue we might introduce a message type to get cluster
state.
But I agree we can just update TopicMetadataResponse to include
controllerId (and probably smth else).
What are you thougths?
Thanks,
Andrii
On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang wangg...@gmail.com
wrote:
I think for the topics commands we can actually merge
create/alter/delete/describe as one request type since their formats are
very much similar, and keep list-topics and others like
partition-reassignment / preferred-leader-election as separate request
types, I also left some other comments on the RB (
https://reviews.apache.org/r/29301/).
On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah I totally agree that we don't want to just have one do admin
stuff
command that has the union of all parameters.
What I am saying is that command line tools are one client of the
administrative apis, but these will be used in
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Jay,
Re error messages: you are right, in most cases client will have enough
context to show descriptive error message. My concern is that we will have
to
add lots of new error codes for each possible error. Of course, we could
reuse
some of existing like UknownTopicOrPartitionCode, but we will also need to
add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both for topic
name and config, and probably user would like to know what exactly
is wrong in his config), InvalidReplicaAssignment, InternalError (e.g.
zookeeper failure) etc.
And this is only for TopicCommand, we will also need to add similar stuff
for
ReassignPartitions, PreferredReplica. So we'll end up with a large list of
error codes, used only in Admin protocol.
Having said that, I agree my proposal is not consistent with other cases.
Maybe we can find better solution or something in-between.
Re Hangout chat: I think it is a great idea. This way we can move on faster.
Let's agree somehow on date/time so people can join. Will work for me this
and
next week almost anytime if agreed in advance.
Thanks,
Andrii
On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps jay.kr...@gmail.com wrote:
Hey Andrii,
Generally we can do good error handling without needing custom server-side
messages. I.e. generally the client has the context to know that if it got
an error that the topic doesn't exist to say Topic X doesn't exist rather
than error code 14 (or whatever). Maybe there are specific cases where
this is hard? If we want to add server-side error messages we really do
need to do this in a consistent way across the protocol.
I still have a bunch of open questions here from my previous list. I will
be out for the next few days for Strata though. Maybe we could do a Google
Hangout chat on any open issues some time towards the end of next week for
anyone interested in this ticket? I have a feeling that might progress
things a little faster than email--I think we could talk through those
issues I brought up fairly quickly...
-Jay
On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
Hi all,
I'm trying to address some of the issues which were mentioned earlier
about
Admin RQ/RP format. One of those was about batching operations. What if
we
follow TopicCommand approach and let people specify topic-name by regexp
-
would that cover most of the use cases?
Secondly, is what information should we generally provide in Admin
responses.
I realize that Admin commands don't imply they will be used only in CLI
but,
it seems to me, CLI is a very important client of this feature. In this
case,
seems logical, we would like to provide users with rich experience in
terms
of
getting results / errors of the executed commands. Usually we supply with
responses only errorCode, which looks very limiting, in case of CLI we
may
want to print human readable error description.
So, taking into account previous item about batching, what do you think
about
having smth like:
('create' doesn't support regexp)
CreateTopicRequest = TopicName Partitions Replicas ReplicaAssignment
[Config]
CreateTopicResponse = ErrorCode ErrorDescription
ErrorCode = int16
ErrorDescription = string (empty if successful)
AlterTopicRequest - TopicNameRegexp Partitions ReplicaAssignment
[AddedConfig] [DeletedConfig]
AlterTopicResponse - [TopicName ErrorCode ErrorDescription]
CommandErrorCode CommandErrorDescription
CommandErrorCode = int16
CommandErrorDescription = string (nonempty in case of fatal error,
e.g.
we couldn't get topics by regexp)
DescribeTopicRequest - TopicNameRegexp
DescribeTopicResponse - [TopicName TopicDescription ErrorCode
ErrorDescription] CommandErrorCode CommandErrorDescription
Also, any thoughts about our discussion regarding re-routing facility? In
my
understanding, it is like between augmenting TopicMetadataRequest
(to include at least controllerId) and implementing new generic
re-routing
facility so sending messages to controller will be handled by it.
Thanks,
Andrii Biletskyi
On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
@Guozhang:
Thanks for your comments, I've answered some of those. The main thing
is
having merged request for create-alter-delete-describe - I have some
concerns about this approach.
@*Jay*:
I see that introduced ClusterMetadaRequest is also one of the concerns.
We
can solve it if we implement re-routing facility. But I agree with
Guozhang - it will make clients' internals a little bit easier but this
seems to be a complex logic to implement and support then. Especially
for
Fetch and Produce (even if we add re-routing later for these requests).
Also people will tend to avoid this re-routing facility and hold local
cluster cache to ensure their high-priority requests (which some of
the
admin requests are) not sent
[jira] [Commented] (KAFKA-1887) controller error message on shutting the last broker
[ https://issues.apache.org/jira/browse/KAFKA-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326455#comment-14326455 ] Helena Edelson commented on KAFKA-1887: --- I see this consistently on shutdown in version 0.8.2.0 controller error message on shutting the last broker Key: KAFKA-1887 URL: https://issues.apache.org/jira/browse/KAFKA-1887 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Sriharsha Chintalapani Priority: Minor Fix For: 0.8.3 We always see the following error in state-change log on shutting down the last broker. [2015-01-20 13:21:04,036] ERROR Controller 0 epoch 3 initiated state change for partition [test,0] from OfflinePartition to OnlinePartition failed (state.change.logger) kafka.common.NoReplicaOnlineException: No replica for partition [test,0] is alive. Live brokers are: [Set()], Assigned replicas are: [List(0)] at kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:75) at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:357) at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:206) at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:120) at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:117) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:117) at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:446) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:373) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1852: -- Attachment: KAFKA-1852_2015-02-18_13:13:17.patch OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Sriharsha Chintalapani Attachments: KAFKA-1852.patch, KAFKA-1852_2015-01-19_10:44:01.patch, KAFKA-1852_2015-02-12_16:46:10.patch, KAFKA-1852_2015-02-16_13:21:46.patch, KAFKA-1852_2015-02-18_13:13:17.patch Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1964) testPartitionReassignmentCallback hangs occasionally
[ https://issues.apache.org/jira/browse/KAFKA-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1964: --- Attachment: stack.out Saw the test hang once in trunk. Attached is the full stacktrace. Test worker prio=5 tid=7ffeb4105000 nid=0x116acd000 runnable [116aca000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked 7f548be50 (a sun.nio.ch.Util$2) - locked 7f548be38 (a java.util.Collections$UnmodifiableSet) - locked 7f5420990 (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.kafka.common.network.Selector.select(Selector.java:375) at org.apache.kafka.common.network.Selector.poll(Selector.java:220) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:209) at org.apache.kafka.clients.consumer.KafkaConsumer.awaitMetadataUpdate(KafkaConsumer.java:956) at org.apache.kafka.clients.consumer.KafkaConsumer.listOffset(KafkaConsumer.java:1353) at org.apache.kafka.clients.consumer.KafkaConsumer.resetOffset(KafkaConsumer.java:1423) at org.apache.kafka.clients.consumer.KafkaConsumer.fetchMissingPositionsOrResetThem(KafkaConsumer.java:1305) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:700) - locked 7f548d990 (a org.apache.kafka.clients.consumer.KafkaConsumer) at kafka.api.ConsumerTest.testPartitionReassignmentCallback(ConsumerTest.scala:239) testPartitionReassignmentCallback hangs occasionally - Key: KAFKA-1964 URL: https://issues.apache.org/jira/browse/KAFKA-1964 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Attachments: stack.out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1952) High CPU Usage in 0.8.2 release
[ https://issues.apache.org/jira/browse/KAFKA-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1952: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review. Committed to 0.8.2.1 and trunk. High CPU Usage in 0.8.2 release --- Key: KAFKA-1952 URL: https://issues.apache.org/jira/browse/KAFKA-1952 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Jun Rao Priority: Critical Fix For: 0.8.2.1 Attachments: kafka-1952.patch, kafka-1952.patch, kafka-1952_2015-02-15_15:26:33.patch Brokers with high partition count see increased CPU usage when migrating from 0.8.1.1 to 0.8.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1964) testPartitionReassignmentCallback hangs occasionally
Jun Rao created KAFKA-1964: -- Summary: testPartitionReassignmentCallback hangs occasionally Key: KAFKA-1964 URL: https://issues.apache.org/jira/browse/KAFKA-1964 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1887) controller error message on shutting the last broker
[ https://issues.apache.org/jira/browse/KAFKA-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326455#comment-14326455 ] Helena Edelson edited comment on KAFKA-1887 at 2/18/15 7:57 PM: I see this consistently on shutdown in version 0.8.2.0. Shutting the controller down first as a workaround works. was (Author: helena_e): I see this consistently on shutdown in version 0.8.2.0 controller error message on shutting the last broker Key: KAFKA-1887 URL: https://issues.apache.org/jira/browse/KAFKA-1887 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Sriharsha Chintalapani Priority: Minor Fix For: 0.8.3 We always see the following error in state-change log on shutting down the last broker. [2015-01-20 13:21:04,036] ERROR Controller 0 epoch 3 initiated state change for partition [test,0] from OfflinePartition to OnlinePartition failed (state.change.logger) kafka.common.NoReplicaOnlineException: No replica for partition [test,0] is alive. Live brokers are: [Set()], Assigned replicas are: [List(0)] at kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:75) at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:357) at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:206) at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:120) at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:117) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:117) at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:446) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:373) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357) at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357) at kafka.utils.Utils$.inLock(Utils.scala:535) at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29912: Patch for KAFKA-1852
On Feb. 13, 2015, 7:01 p.m., Joel Koshy wrote: core/src/main/scala/kafka/server/OffsetManager.scala, line 215 https://reviews.apache.org/r/29912/diff/3/?file=862699#file862699line215 Minor comment. I think this may be better to pass in to the OffsetManager. We should even use it in loadOffsets to discard offsets that are from topics that have been deleted. We can do that in a separate jira - I don't think our handling for clearing out offsets on a delete topic is done yet - Onur Karaman did it for ZK based offsets but we need a separate jira to delete Kafka-based offsets. Sriharsha Chintalapani wrote: Thanks for the review. Since offsetmanager initialized in KafkaServer and metadataCache in KafkaApis , in the latest patch I added setMetadataCache in OffsetManager and calling it in KafkaApis. Please take a look Joel Koshy wrote: In that case I think it is just better to create the cache outside (in KafkaServer and pass it in to KafkaApis). The metadataCache is useful enough to be used in other places (other than just KafkaApis). Updated the patch as per your suggestion. Please take a look. thanks. - Sriharsha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29912/#review72413 --- On Feb. 18, 2015, 9:13 p.m., Sriharsha Chintalapani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29912/ --- (Updated Feb. 18, 2015, 9:13 p.m.) Review request for kafka. Bugs: KAFKA-1852 https://issues.apache.org/jira/browse/KAFKA-1852 Repository: kafka Description --- KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. Added contains method to MetadataCache. KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. Diffs - core/src/main/scala/kafka/server/KafkaApis.scala 703886a1d48e6d2271da67f8b89514a6950278dd core/src/main/scala/kafka/server/KafkaServer.scala 7e5ddcb9be8fcef3df6ebc82a13ef44ef95f73ae core/src/main/scala/kafka/server/MetadataCache.scala 4c70aa7e0157b85de5e24736ebf487239c4571d0 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 Diff: https://reviews.apache.org/r/29912/diff/ Testing --- Thanks, Sriharsha Chintalapani
[jira] [Commented] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326566#comment-14326566 ] Sriharsha Chintalapani commented on KAFKA-1852: --- Updated reviewboard https://reviews.apache.org/r/29912/diff/ against branch origin/trunk OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Assignee: Sriharsha Chintalapani Attachments: KAFKA-1852.patch, KAFKA-1852_2015-01-19_10:44:01.patch, KAFKA-1852_2015-02-12_16:46:10.patch, KAFKA-1852_2015-02-16_13:21:46.patch, KAFKA-1852_2015-02-18_13:13:17.patch Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29912: Patch for KAFKA-1852
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29912/ --- (Updated Feb. 18, 2015, 9:13 p.m.) Review request for kafka. Bugs: KAFKA-1852 https://issues.apache.org/jira/browse/KAFKA-1852 Repository: kafka Description (updated) --- KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. Added contains method to MetadataCache. KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. KAFKA-1852. OffsetCommitRequest can commit offset on unknown topic. Diffs (updated) - core/src/main/scala/kafka/server/KafkaApis.scala 703886a1d48e6d2271da67f8b89514a6950278dd core/src/main/scala/kafka/server/KafkaServer.scala 7e5ddcb9be8fcef3df6ebc82a13ef44ef95f73ae core/src/main/scala/kafka/server/MetadataCache.scala 4c70aa7e0157b85de5e24736ebf487239c4571d0 core/src/main/scala/kafka/server/OffsetManager.scala 83d52643028c5628057dc0aa29819becfda61fdb core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 Diff: https://reviews.apache.org/r/29912/diff/ Testing --- Thanks, Sriharsha Chintalapani
[jira] [Created] (KAFKA-1965) Leaner DelayedItem
Yasuhiro Matsuda created KAFKA-1965: --- Summary: Leaner DelayedItem Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30570: Patch for KAFKA-1914
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30570/#review73020 --- Thanks for the patch. A couple of comments. 1. We already measure the total produce/fetch request rate in RequestMetrics. Now, we are duplicating that in BrokerTopics. Is there a way to avoid the duplication? 2. The following unit test fails consistently for me, when running all unit tests. It does pass when running individually. kafka.server.SimpleFetchTest testReadFromLog FAILED junit.framework.AssertionFailedError: Counts should increment after fetch expected:5 but was:51432 at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:277) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:130) at kafka.server.SimpleFetchTest.testReadFromLog(SimpleFetchTest.scala:145) - Jun Rao On Feb. 17, 2015, 11:46 p.m., Aditya Auradkar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30570/ --- (Updated Feb. 17, 2015, 11:46 p.m.) Review request for kafka and Joel Koshy. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- BrokerTopicStats changes to aggregate on a per-topic basis the total fetch and produce requests Diffs - core/src/main/scala/kafka/server/KafkaRequestHandler.scala e4053fbe8ef78bf8bc39cb3f8ea4c21032613a16 core/src/main/scala/kafka/server/ReplicaManager.scala fb948b9ab28c516e81dab14dcbe211dcd99842b6 core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala ccf5e2e36260b2484181b81d1b06e81de972674b Diff: https://reviews.apache.org/r/30570/diff/ Testing --- I've added asserts to the SimpleFetchTest to count the number of fetch requests. I'm going to file an additional jira to add unit tests for all the BrokerTopicMetrics updated via ReplicaManager Thanks, Aditya Auradkar
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Status: Patch Available (was: Open) Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Description: In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. (was: In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. ) Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial Attachments: KAFKA-1965.patch In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30570: Patch for KAFKA-1914
On Feb. 18, 2015, 10:16 p.m., Jun Rao wrote: Thanks for the patch. A couple of comments. 1. We already measure the total produce/fetch request rate in RequestMetrics. Now, we are duplicating that in BrokerTopics. Is there a way to avoid the duplication? 2. The following unit test fails consistently for me, when running all unit tests. It does pass when running individually. kafka.server.SimpleFetchTest testReadFromLog FAILED junit.framework.AssertionFailedError: Counts should increment after fetch expected:5 but was:51432 at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:277) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:130) at kafka.server.SimpleFetchTest.testReadFromLog(SimpleFetchTest.scala:145) For (1) I noted this on the review but it is convenient to have the total under BrokerTopicMetrics as well which is a reasonable tradeoff since it only adds two additional meters. For (2) Aditya, can you look into that? - Joel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30570/#review73020 --- On Feb. 17, 2015, 11:46 p.m., Aditya Auradkar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30570/ --- (Updated Feb. 17, 2015, 11:46 p.m.) Review request for kafka and Joel Koshy. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- BrokerTopicStats changes to aggregate on a per-topic basis the total fetch and produce requests Diffs - core/src/main/scala/kafka/server/KafkaRequestHandler.scala e4053fbe8ef78bf8bc39cb3f8ea4c21032613a16 core/src/main/scala/kafka/server/ReplicaManager.scala fb948b9ab28c516e81dab14dcbe211dcd99842b6 core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala ccf5e2e36260b2484181b81d1b06e81de972674b Diff: https://reviews.apache.org/r/30570/diff/ Testing --- I've added asserts to the SimpleFetchTest to count the number of fetch requests. I'm going to file an additional jira to add unit tests for all the BrokerTopicMetrics updated via ReplicaManager Thanks, Aditya Auradkar
[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available
[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326657#comment-14326657 ] Parth Brahmbhatt commented on KAFKA-1788: - Can someone from the kafka team respond to the questions asked above. producer record can stay in RecordAccumulator forever if leader is no available --- Key: KAFKA-1788 URL: https://issues.apache.org/jira/browse/KAFKA-1788 Project: Kafka Issue Type: Bug Components: core, producer Affects Versions: 0.8.2.0 Reporter: Jun Rao Assignee: Jun Rao Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1788.patch, KAFKA-1788_2015-01-06_13:42:37.patch, KAFKA-1788_2015-01-06_13:44:41.patch In the new producer, when a partition has no leader for a long time (e.g., all replicas are down), the records for that partition will stay in the RecordAccumulator until the leader is available. This may cause the bufferpool to be full and the callback for the produced message to block for a long time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Description: In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Attachment: KAFKA-1965.patch Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial Attachments: KAFKA-1965.patch In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Status: Patch Available (was: Open) Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial Attachments: KAFKA-1965.patch In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-1965: Status: Open (was: Patch Available) Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] 0.8.2.1 Candidate 1
+1 On Feb 18, 2015, at 7:56 PM, Jun Rao j...@confluent.io wrote: This is the first candidate for release of Apache Kafka 0.8.2.1. This only fixes one critical issue (KAFKA-1952) in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Saturday, Feb 21, 7pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=c1b4c58531343dce80232e0122d085fc687633f6 /*** Thanks, Jun
Re: [VOTE] 0.8.2.1 Candidate 1
+1 On Feb 18, 2015 7:23 PM, Matt Narrell matt.narr...@gmail.com wrote: +1 On Feb 18, 2015, at 7:56 PM, Jun Rao j...@confluent.io wrote: This is the first candidate for release of Apache Kafka 0.8.2.1. This only fixes one critical issue (KAFKA-1952) in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Saturday, Feb 21, 7pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=c1b4c58531343dce80232e0122d085fc687633f6 /*** Thanks, Jun
[jira] [Commented] (KAFKA-1965) Leaner DelayedItem
[ https://issues.apache.org/jira/browse/KAFKA-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326973#comment-14326973 ] Sriharsha Chintalapani commented on KAFKA-1965: --- [~yasuhiro.matsuda] Changes looks good to me. Can you attach it to the review board here are instructions https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review Leaner DelayedItem -- Key: KAFKA-1965 URL: https://issues.apache.org/jira/browse/KAFKA-1965 Project: Kafka Issue Type: Improvement Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Joel Koshy Priority: Trivial Attachments: KAFKA-1965.patch In DelayedItem, which is a superclass of DelayedOperation, both the creation timestamp and the length delay are stored. However, all it needs is one timestamp that is the due of the item. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29301: Patch for KAFKA-1694
On Feb. 3, 2015, 7:14 p.m., Guozhang Wang wrote:
core/src/main/scala/kafka/server/TopicCommandHelper.scala, lines 1-17
https://reviews.apache.org/r/29301/diff/7/?file=821380#file821380line1
One general comment:
For some topic commands, why use AdminUtils to write ZK path again
instead of handle it via the controller directly? Or this is still WIP?
Andrii Biletskyi wrote:
Not sure I understand you. You mean technially calling ZK client from
Controller class, not through TopicCommandHelper? If so - it's just to leave
KafkaApi clean and small.
Guozhang Wang wrote:
For example, upon receiving a create-topic request, the helper class will
call AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK() which will
just write this request to ZK admin path for it to be captured by controller;
however since only the broker with the active controller will receive such
requests why don't we just hand off the request from KafkaApi to the
controller to handle it.
One question, though, is that we need to make sure concurrency is correct
for controller handling multiple such tasks, and we have some thoughts about
how to deal with such cases (see Jiangjie and my commnets in KAFKA-1305).
Thanks for explanation.
So instead of current workflow:
CreateTopicRequest - Helper class - AdminUtils - zk path is created -
Controller's changeTopicListener picks up the change - topic is created
You propose:
CreateTopicRequest - Controller directly executes logic from
ChangeTopicListener
?
Very interesting idea! Can we make a separate ticket for that? I tried to port
TopicCommand as is in order to have at least for now working end-to-end
infrastructure to handle Admin commands. I believe this is more like
refactoring TopicCommand (probably delete- and alterTopic should be changed
too). I'm a bit concerned adding this refactoring will require additional
efforts to test (especially taking into account your note about KAFKA-1305) and
time to agree on approach we will use to address this issue.
On Feb. 3, 2015, 7:14 p.m., Guozhang Wang wrote:
core/src/main/scala/kafka/controller/ControllerChannelManager.scala, lines
301-310
https://reviews.apache.org/r/29301/diff/7/?file=821376#file821376line301
Do not understand the rationale behind this: could you add some
comments? Particularly, why we want to send an empty metadata map to the
brokers with forceSendBrokerInfo?
Andrii Biletskyi wrote:
Thanks, this is done because on startup we don't send UpdateMetadaRequest
(updateMetadataRequestMap is empty) and thus brokers' cache is not filled
with brokers and controller. This leads to ClusterMetadataRequest can't be
served correctly.
I'm not sure this is the best way to do it, open for suggestions.
Guozhang Wang wrote:
In this case can we just use addUpdateMetadataRequestForBrokers() before
calling sendRequestsToBrokers()?
If I understood correctly - addUpdateMetadataRequestForBrokers() is already
called, it's just nothing is added to UpdateMetadata. The steps are the
following:
1. One broker cluster is started (no topics)
2. KafkaController.onControllerFailover() is called
3. sendUpdateMetadataRequest()
4. addUpdateMetadataRequest(): updateMetadataRequest is created foreach
controllerContext.partitionLeadershipInfo.keySet (which is empty)
5. sendRequestsToBrokers(): we send UpdateMetadata foreach broker from
updateMetadataRequestMap (which is empty) - broker holding a controller's role
doesn't receive UpdateMetadataRequest
So essentially the problem is that UpdateMetadaRequest holds data about
controller, brokers _and_ partitionState but we send UpdateMetadaRequest only
if there is partitionState update to be sent.
On Feb. 3, 2015, 7:14 p.m., Guozhang Wang wrote:
clients/src/main/java/org/apache/kafka/common/requests/admin/AbstractAdminRequest.java,
lines 1-28
https://reviews.apache.org/r/29301/diff/7/?file=821321#file821321line1
Wondering if an abstract admin request is necessary, as it does not
have many common interface functions.
Andrii Biletskyi wrote:
This is needed to avoid code dupliaction in admin clients. See
RequestDispatcher for example.
You will need to call admin request and get response of that type. Having
AbstractAdminRequest (specifically createResponseCounterpart) lets you have:
```
public T extends AbstractAdminResponse T
sendAdminRequest(AbstractAdminRequestT abstractRequest) throws Exception {
```
Instead of sendCreateTopicRequest, sendAlter... etc. If there is a better
and cleaner way to achive this - please let me know.
Guozhang Wang wrote:
I see. How about changing sendAdminRequest(AbstractAdminRequestT) to
sendRequest(ClientRequest) and the caller like AlterTopicCommand.execute()
will be:
AlterTopicRequest alterTopicRequest = // create the request
ClientRequest request = new
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Hi all,
I'm trying to address some of the issues which were mentioned earlier about
Admin RQ/RP format. One of those was about batching operations. What if we
follow TopicCommand approach and let people specify topic-name by regexp -
would that cover most of the use cases?
Secondly, is what information should we generally provide in Admin
responses.
I realize that Admin commands don't imply they will be used only in CLI
but,
it seems to me, CLI is a very important client of this feature. In this
case,
seems logical, we would like to provide users with rich experience in terms
of
getting results / errors of the executed commands. Usually we supply with
responses only errorCode, which looks very limiting, in case of CLI we may
want to print human readable error description.
So, taking into account previous item about batching, what do you think
about
having smth like:
('create' doesn't support regexp)
CreateTopicRequest = TopicName Partitions Replicas ReplicaAssignment
[Config]
CreateTopicResponse = ErrorCode ErrorDescription
ErrorCode = int16
ErrorDescription = string (empty if successful)
AlterTopicRequest - TopicNameRegexp Partitions ReplicaAssignment
[AddedConfig] [DeletedConfig]
AlterTopicResponse - [TopicName ErrorCode ErrorDescription]
CommandErrorCode CommandErrorDescription
CommandErrorCode = int16
CommandErrorDescription = string (nonempty in case of fatal error, e.g.
we couldn't get topics by regexp)
DescribeTopicRequest - TopicNameRegexp
DescribeTopicResponse - [TopicName TopicDescription ErrorCode
ErrorDescription] CommandErrorCode CommandErrorDescription
Also, any thoughts about our discussion regarding re-routing facility? In
my
understanding, it is like between augmenting TopicMetadataRequest
(to include at least controllerId) and implementing new generic re-routing
facility so sending messages to controller will be handled by it.
Thanks,
Andrii Biletskyi
On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
@Guozhang:
Thanks for your comments, I've answered some of those. The main thing is
having merged request for create-alter-delete-describe - I have some
concerns about this approach.
@*Jay*:
I see that introduced ClusterMetadaRequest is also one of the concerns. We
can solve it if we implement re-routing facility. But I agree with
Guozhang - it will make clients' internals a little bit easier but this
seems to be a complex logic to implement and support then. Especially for
Fetch and Produce (even if we add re-routing later for these requests).
Also people will tend to avoid this re-routing facility and hold local
cluster cache to ensure their high-priority requests (which some of the
admin requests are) not sent to some busy broker where they wait to be
routed to the correct one.
As pointed out by Jun here (
https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530)
to solve the issue we might introduce a message type to get cluster state.
But I agree we can just update TopicMetadataResponse to include
controllerId (and probably smth else).
What are you thougths?
Thanks,
Andrii
On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang wangg...@gmail.com wrote:
I think for the topics commands we can actually merge
create/alter/delete/describe as one request type since their formats are
very much similar, and keep list-topics and others like
partition-reassignment / preferred-leader-election as separate request
types, I also left some other comments on the RB (
https://reviews.apache.org/r/29301/).
On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah I totally agree that we don't want to just have one do admin
stuff
command that has the union of all parameters.
What I am saying is that command line tools are one client of the
administrative apis, but these will be used in a number of scenarios so
they should make logical sense even in the absence of the command line
tool. Hence comments like trying to clarify the relationship between
ClusterMetadata and TopicMetadata...these kinds of things really need
to be
thought through.
Hope that makes sense.
-Jay
On Wed, Feb 11, 2015 at 1:41 PM, Andrii Biletskyi
andrii.bilets...@stealth.ly wrote:
Jay,
Thanks for answering. You understood correctly, most of my comments
were
related to your point 1) - about well thought-out apis. Also, yes,
as I
understood we would like to introduce a single unified CLI tool with
centralized server-side request handling for lots of existing ones
(incl.
TopicCommand, CommitOffsetChecker, ReassignPartitions, smth else if
added
in future). In our previous discussion (
https://issues.apache.org/jira/browse/KAFKA-1694) people said they'd
rather
have a separate message for each command, so, yes, this way I came to
1-1
mapping between commands in the
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326171#comment-14326171 ] Andrii Biletskyi commented on KAFKA-1694: - [~guozhang], yes, initially I did split it to several patches. The problem is that almost all tickets depend on each other. Also I believe that single patch lets you better understand the whole picture (and the use case - CLI) so people can comment, argue in mail list. But I totally agree with you this way we can miss some hidden issues. I would prefer to collect some feedback to be sure we more or less agree on approach and after that I can split this big feature to sub-patches. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Biletskyi updated KAFKA-1694: Comment: was deleted (was: [~guozhang], yes, initially I did split it to several patches. The problem is that almost all tickets depend on each other. Also I believe that single patch lets you better understand the whole picture (and the use case - CLI) so people can comment, argue in mail list. But I totally agree with you this way we can miss some hidden issues. I would prefer to collect some feedback to be sure we more or less agree on approach and after that I can split this big feature to sub-patches. ) kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326172#comment-14326172 ] Andrii Biletskyi commented on KAFKA-1694: - [~guozhang], yes, initially I did split it to several patches. The problem is that almost all tickets depend on each other. Also I believe that single patch lets you better understand the whole picture (and the use case - CLI) so people can comment, argue in mail list. But I totally agree with you this way we can miss some hidden issues. I would prefer to collect some feedback to be sure we more or less agree on approach and after that I can split this big feature to sub-patches. kafka command line and centralized operations - Key: KAFKA-1694 URL: https://issues.apache.org/jira/browse/KAFKA-1694 Project: Kafka Issue Type: Bug Reporter: Joe Stein Assignee: Andrii Biletskyi Priority: Critical Fix For: 0.8.3 Attachments: KAFKA-1694.patch, KAFKA-1694_2014-12-24_21:21:51.patch, KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1772_1802_1775_1774_v2.patch https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Kafka-trunk #396
See https://builds.apache.org/job/Kafka-trunk/396/changes Changes: [junrao] kafka-1952; High CPU Usage in 0.8.2 release; patched by Jun Rao; reviewed by Guozhang Wang, Ewen Cheslack-Postava and Neha Narkhede [jjkoshy] KAFKA-1914; follow-up to address unit test failure -- [...truncated 2707 lines...] at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerSendTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerSendTest.scala:39) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerSendTest.setUp(ProducerSendTest.scala:55) kafka.api.test.ProducerFailureHandlingTest testNotEnoughReplicas FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:74) kafka.api.test.ProducerFailureHandlingTest testInvalidPartition FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:74) kafka.api.test.ProducerFailureHandlingTest testTooLargeRecordWithAckZero FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:74) kafka.api.test.ProducerFailureHandlingTest testTooLargeRecordWithAckOne FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) at kafka.zk.EmbeddedZookeeper.init(EmbeddedZookeeper.scala:33) at kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33) at kafka.api.test.ProducerFailureHandlingTest.kafka$integration$KafkaServerTestHarness$$super$setUp(ProducerFailureHandlingTest.scala:38) at kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:44) at kafka.api.test.ProducerFailureHandlingTest.setUp(ProducerFailureHandlingTest.scala:74) kafka.api.test.ProducerFailureHandlingTest testNonExistentTopic FAILED java.net.BindException: Address already in use at sun.nio.ch.Net.bind(Native Method) at
[VOTE] KIP-8: Add a flush() method to the new producer
https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API +1 binding -Jay
Re: Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/#review73057 --- core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala https://reviews.apache.org/r/31169/#comment119242 Should we remove the comment? - Jun Rao On Feb. 19, 2015, 1:30 a.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 19, 2015, 1:30 a.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Allow constructing explicitly versioned offset fetch request in javaapi Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Build failed in Jenkins: Kafka-trunk #397
See https://builds.apache.org/job/Kafka-trunk/397/changes Changes: [jjkoshy] KAFKA-1914; follow-up to fix SimpleFetchTest; reviewed by Joel Koshy -- [...truncated 1269 lines...] kafka.producer.AsyncProducerTest testPartitionAndCollateEvents PASSED kafka.producer.AsyncProducerTest testSerializeEvents PASSED kafka.producer.AsyncProducerTest testInvalidPartition PASSED kafka.producer.AsyncProducerTest testNoBroker PASSED kafka.producer.AsyncProducerTest testIncompatibleEncoder PASSED kafka.producer.AsyncProducerTest testRandomPartitioner PASSED kafka.producer.AsyncProducerTest testFailedSendRetryLogic PASSED kafka.producer.AsyncProducerTest testJavaProducer PASSED kafka.producer.AsyncProducerTest testInvalidConfiguration PASSED kafka.log.CleanerTest testCleanSegments PASSED kafka.log.CleanerTest testCleaningWithDeletes PASSED kafka.log.CleanerTest testCleanSegmentsWithAbort PASSED kafka.log.CleanerTest testSegmentGrouping PASSED kafka.log.CleanerTest testBuildOffsetMap PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[1] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[2] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[3] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[4] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[5] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[6] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[7] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[8] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[9] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[10] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[11] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[12] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[13] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[14] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[15] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[16] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[17] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[18] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[19] PASSED kafka.log.LogManagerTest testCreateLog PASSED kafka.log.LogManagerTest testGetNonExistentLog PASSED kafka.log.LogManagerTest testCleanupExpiredSegments PASSED kafka.log.LogManagerTest testCleanupSegmentsToMaintainSize PASSED kafka.log.LogManagerTest testTimeBasedFlush PASSED kafka.log.LogManagerTest testLeastLoadedAssignment PASSED kafka.log.LogManagerTest testTwoLogManagersUsingSameDirFails PASSED kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithTrailingSlash PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.log.LogConfigTest testFromPropsDefaults PASSED kafka.log.LogConfigTest testFromPropsEmpty PASSED kafka.log.LogConfigTest testFromPropsToProps PASSED kafka.log.LogConfigTest testFromPropsInvalid PASSED kafka.log.OffsetIndexTest truncate PASSED kafka.log.OffsetIndexTest randomLookupTest PASSED kafka.log.OffsetIndexTest lookupExtremeCases PASSED kafka.log.OffsetIndexTest appendTooMany PASSED kafka.log.OffsetIndexTest appendOutOfOrder PASSED kafka.log.OffsetIndexTest testReopen PASSED kafka.log.FileMessageSetTest testWrittenEqualsRead PASSED kafka.log.FileMessageSetTest testIteratorIsConsistent PASSED kafka.log.FileMessageSetTest testSizeInBytes PASSED kafka.log.FileMessageSetTest testWriteTo PASSED kafka.log.FileMessageSetTest testFileSize PASSED kafka.log.FileMessageSetTest testIterationOverPartialAndTruncation PASSED kafka.log.FileMessageSetTest testIterationDoesntChangePosition PASSED kafka.log.FileMessageSetTest testRead PASSED kafka.log.FileMessageSetTest testSearch PASSED kafka.log.FileMessageSetTest testIteratorWithLimits PASSED kafka.log.FileMessageSetTest testTruncate PASSED kafka.log.LogCleanerIntegrationTest cleanerTest PASSED kafka.log.OffsetMapTest testBasicValidation PASSED kafka.log.OffsetMapTest testClear PASSED kafka.log.LogTest testTimeBasedLogRoll PASSED kafka.log.LogTest testTimeBasedLogRollJitter PASSED kafka.log.LogTest testSizeBasedLogRoll PASSED kafka.log.LogTest testLoadEmptyLog PASSED kafka.log.LogTest testAppendAndReadWithSequentialOffsets PASSED kafka.log.LogTest testAppendAndReadWithNonSequentialOffsets PASSED kafka.log.LogTest testReadAtLogGap PASSED kafka.log.LogTest testReadOutOfRange PASSED kafka.log.LogTest testLogRolls PASSED kafka.log.LogTest testCompressedMessages PASSED kafka.log.LogTest testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED
Re: Review Request 31168: Patch for KAFKA-1914
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31168/ --- (Updated Feb. 19, 2015, 12:01 a.m.) Review request for kafka. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Fixing failing unit test Fixing failing unit test Diffs - core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 525c835803b048c952667567cd205b4b06824391 Diff: https://reviews.apache.org/r/31168/diff/ Testing (updated) --- All unit tests now pass. BUILD SUCCESSFUL Total time: 9 mins 35.296 secs Thanks, Aditya Auradkar
Re: Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/#review73060 --- Ship it! Ship It! - Jun Rao On Feb. 19, 2015, 1:30 a.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 19, 2015, 1:30 a.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Allow constructing explicitly versioned offset fetch request in javaapi Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Re: [VOTE] KIP-8: Add a flush() method to the new producer
+1 binding ~ Joestein On Feb 18, 2015 6:50 PM, Jay Kreps j...@confluent.io wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API +1 binding -Jay
[jira] [Assigned] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar reassigned KAFKA-1546: -- Assignee: Aditya Auradkar (was: nicu marasoiu) Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Assignee: Aditya Auradkar Labels: newbie++ Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326773#comment-14326773 ] Aditya Auradkar commented on KAFKA-1546: [~nmarasoi] I'm going to assign this to myself since I haven't heard back. Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Assignee: nicu marasoiu Labels: newbie++ Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer
Actually, could you clarify this a bit (since I'm not sure which thread you are referring to) - specifically, how would this tie in with the current timeout we have for the producer (for example)? On Tue, Feb 17, 2015 at 02:55:44PM -0800, Jay Kreps wrote: Yeah there was a separate thread on adding a client-side timeout to requests. We should have this in the new java clients, it just isn't there yet. When we do this the flush() call will implicitly have the same timeout as the requests (since they will complete or fail by then). I think this makes flush(timeout) and potentially close(timeout) both unnecessary. -Jay On Tue, Feb 17, 2015 at 2:44 PM, Guozhang Wang wangg...@gmail.com wrote: In the scala clients we have the socket.timeout config as we are using blocking IOs, when such timeout is reached the TimeoutException will be thrown from the socket and the client can handle it accordingly; in the java clients we are switching to non-blocking IOs and hence we will not have the socket timeout any more. I agree that we could add this client request timeout back in the java clients, in addition to allowing client / server's non-blocking selector to close idle sockets. Guozhang On Tue, Feb 17, 2015 at 1:55 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: I'm thinking the flush call timeout will naturally be the timeout for a produce request, No? Currently it seems we don¹t have a timeout for client requests, should we have one? ‹Jiangjie (Becket) Qin On 2/16/15, 8:19 PM, Jay Kreps jay.kr...@gmail.com wrote: Yes, I think we all agree it would be good to add a client-side request timeout. That would effectively imply a flush timeout as well since any requests that couldn't complete in that time would be errors and hence completed in the definition we gave. -Jay On Mon, Feb 16, 2015 at 7:57 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi All, Thanks Jay and all address concern. I am fine with just having flush() method as long as it covers failure mode and resiliency. e.g We had situation where entire Kafka cluster brokers were reachable, but upon adding new kafka node and admin migrated leader to new brokers that new brokers is NOT reachable from producer stand point due to fire wall but metadata would continue to elect new broker as leader for that partition. All I am asking is either you will have to give-up sending to this broker or do something in this scenario. As for the current code 0.8.2 release, caller thread of flush() or close() method would be blocked for ever so all I am asking is https://issues.apache.org/jira/browse/KAFKA-1659 https://issues.apache.org/jira/browse/KAFKA-1660 Also, I recall that there is timeout also added to batch to indicate how long message can retain in memory before expiring. Given, all this should this API be consistent with others up coming patches for addressing similar problem(s). Otherwise, what we have done is spawn a thread for just calling close() or flush with timeout for join on caller end. Anyway, I just wanted to give you issues with existing API and if you guys think this is fine then, I am ok with this approach. It is just that caller will have to do bit more work. Thanks, Bhavesh On Thursday, February 12, 2015, Joel Koshy jjkosh...@gmail.com wrote: Yes that is a counter-example. I'm okay either way on whether we should have just flush() or have a timeout. Bhavesh, does Jay's explanation a few replies prior address your concern? If so, shall we consider this closed? On Tue, Feb 10, 2015 at 01:36:23PM -0800, Jay Kreps wrote: Yeah we could do that, I guess I just feel like it adds confusion because then you have to think about which timeout you want, when likely you don't want a timeout at all. I guess the pattern I was thinking of was fflush or the java equivalent, which don't have timeouts: http://docs.oracle.com/javase/7/docs/api/java/io/OutputStream.html#flush( ) -Jay On Tue, Feb 10, 2015 at 10:41 AM, Joel Koshy jjkosh...@gmail.com wrote: I think tryFlush with a timeout sounds good to me. This is really more for consistency than anything else. I cannot think of any standard blocking calls off the top of my head that don't have a timed variant. E.g., Thread.join, Object.wait, Future.get Either that, or they provide an entirely non-blocking mode (e.g., socketChannel.connect followed by finishConnect) Thanks, Joel On Tue, Feb 10, 2015 at 11:30:47AM -0500, Joe Stein wrote: Jay, The .flush()
Build failed in Jenkins: KafkaPreCommit #10
See https://builds.apache.org/job/KafkaPreCommit/10/changes Changes: [jjkoshy] KAFKA-1914; follow-up to fix SimpleFetchTest; reviewed by Joel Koshy -- [...truncated 314 lines...] kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithTrailingSlash PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[1] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[2] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[3] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[4] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[5] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[6] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[7] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[8] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[9] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[10] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[11] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[12] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[13] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[14] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[15] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[16] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[17] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[18] PASSED kafka.log.BrokerCompressionTest testBrokerSideCompression[19] PASSED kafka.log.CleanerTest testCleanSegments PASSED kafka.log.CleanerTest testCleaningWithDeletes PASSED kafka.log.CleanerTest testCleanSegmentsWithAbort PASSED kafka.log.CleanerTest testSegmentGrouping PASSED kafka.log.CleanerTest testBuildOffsetMap PASSED kafka.log.OffsetMapTest testBasicValidation PASSED kafka.log.OffsetMapTest testClear PASSED kafka.log.FileMessageSetTest testWrittenEqualsRead PASSED kafka.log.FileMessageSetTest testIteratorIsConsistent PASSED kafka.log.FileMessageSetTest testSizeInBytes PASSED kafka.log.FileMessageSetTest testWriteTo PASSED kafka.log.FileMessageSetTest testFileSize PASSED kafka.log.FileMessageSetTest testIterationOverPartialAndTruncation PASSED kafka.log.FileMessageSetTest testIterationDoesntChangePosition PASSED kafka.log.FileMessageSetTest testRead PASSED kafka.log.FileMessageSetTest testSearch PASSED kafka.log.FileMessageSetTest testIteratorWithLimits PASSED kafka.log.FileMessageSetTest testTruncate PASSED kafka.log.LogCleanerIntegrationTest cleanerTest PASSED kafka.log.LogSegmentTest testTruncate PASSED kafka.log.LogSegmentTest testReadOnEmptySegment PASSED kafka.log.LogSegmentTest testReadBeforeFirstOffset PASSED kafka.log.LogSegmentTest testMaxOffset PASSED kafka.log.LogSegmentTest testReadAfterLast PASSED kafka.log.LogSegmentTest testReadFromGap PASSED kafka.log.LogSegmentTest testTruncateFull PASSED kafka.log.LogSegmentTest testNextOffsetCalculation PASSED kafka.log.LogSegmentTest testChangeFileSuffixes PASSED kafka.log.LogSegmentTest testRecoveryFixesCorruptIndex PASSED kafka.log.LogSegmentTest testRecoveryWithCorruptMessage PASSED kafka.producer.SyncProducerTest testReachableServer PASSED kafka.producer.SyncProducerTest testEmptyProduceRequest PASSED kafka.producer.SyncProducerTest testMessageSizeTooLarge PASSED kafka.producer.SyncProducerTest testMessageSizeTooLargeWithAckZero PASSED kafka.producer.SyncProducerTest testProduceCorrectlyReceivesResponse PASSED kafka.producer.SyncProducerTest testProducerCanTimeout PASSED kafka.producer.SyncProducerTest testProduceRequestWithNoResponse PASSED kafka.producer.SyncProducerTest testNotEnoughReplicas PASSED kafka.producer.AsyncProducerTest testProducerQueueSize PASSED kafka.producer.AsyncProducerTest testProduceAfterClosed PASSED kafka.producer.AsyncProducerTest testBatchSize PASSED kafka.producer.AsyncProducerTest testQueueTimeExpired PASSED kafka.producer.AsyncProducerTest testPartitionAndCollateEvents PASSED kafka.producer.AsyncProducerTest testSerializeEvents PASSED kafka.producer.AsyncProducerTest testInvalidPartition PASSED kafka.producer.AsyncProducerTest testNoBroker PASSED kafka.producer.AsyncProducerTest testIncompatibleEncoder PASSED kafka.producer.AsyncProducerTest testRandomPartitioner PASSED kafka.producer.AsyncProducerTest testFailedSendRetryLogic PASSED kafka.producer.AsyncProducerTest testJavaProducer PASSED kafka.producer.AsyncProducerTest testInvalidConfiguration PASSED kafka.producer.ProducerTest testUpdateBrokerPartitionInfo PASSED kafka.producer.ProducerTest
Jenkins build is back to normal : Kafka-trunk #398
See https://builds.apache.org/job/Kafka-trunk/398/changes
[VOTE] 0.8.2.1 Candidate 1
This is the first candidate for release of Apache Kafka 0.8.2.1. This only fixes one critical issue (KAFKA-1952) in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Saturday, Feb 21, 7pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=c1b4c58531343dce80232e0122d085fc687633f6 /*** Thanks, Jun
Re: Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/#review73056 --- core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala https://reviews.apache.org/r/31169/#comment119238 ack - yes. thanks for catching that. - Joel Koshy On Feb. 18, 2015, 11:55 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 18, 2015, 11:55 p.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Add constructor to javaapi to allow constructing explicitly versioned offset fetch request Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Review Request 31168: Patch for KAFKA-1914
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31168/ --- Review request for kafka. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Fixing failing unit test Fixing failing unit test Diffs - core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 525c835803b048c952667567cd205b4b06824391 Diff: https://reviews.apache.org/r/31168/diff/ Testing --- Thanks, Aditya Auradkar
Re: Review Request 31168: Patch for KAFKA-1914
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31168/ --- (Updated Feb. 18, 2015, 11:55 p.m.) Review request for kafka. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Fixing failing unit test Fixing failing unit test Diffs - core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 525c835803b048c952667567cd205b4b06824391 Diff: https://reviews.apache.org/r/31168/diff/ Testing (updated) --- All unit tests now pass. Thanks, Aditya Auradkar
[jira] [Updated] (KAFKA-1729) add doc for Kafka-based offset management in 0.8.2
[ https://issues.apache.org/jira/browse/KAFKA-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1729: -- Attachment: KAFKA-1729.patch add doc for Kafka-based offset management in 0.8.2 -- Key: KAFKA-1729 URL: https://issues.apache.org/jira/browse/KAFKA-1729 Project: Kafka Issue Type: Sub-task Reporter: Jun Rao Assignee: Joel Koshy Fix For: 0.8.2.0 Attachments: KAFKA-1729.patch, KAFKA-1729.patch, KAFKA-1782-doc-v1.patch, KAFKA-1782-doc-v2.patch, KAFKA-1782-doc-v3.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Add constructor to javaapi to allow constructing explicitly versioned offset fetch request Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Re: Review Request 31169: Patch for KAFKA-1729
On Feb. 19, 2015, 1:32 a.m., Jun Rao wrote: core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala, line 47 https://reviews.apache.org/r/31169/diff/2/?file=868598#file868598line47 Should we remove the comment? Thanks - yes will do that on check-in if everything else looks good to you. - Joel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/#review73057 --- On Feb. 19, 2015, 1:30 a.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 19, 2015, 1:30 a.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Allow constructing explicitly versioned offset fetch request in javaapi Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Re: Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/#review73053 --- core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala https://reviews.apache.org/r/31169/#comment119227 Do you need to fix this too? - Jun Rao On Feb. 18, 2015, 11:55 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 18, 2015, 11:55 p.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Add constructor to javaapi to allow constructing explicitly versioned offset fetch request Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Review Request 31174: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31174/ --- Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description --- Add constructor to javaapi to allow constructing explicitly versioned offset fetch request Diffs - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31174/diff/ Testing --- Thanks, Joel Koshy
[jira] [Commented] (KAFKA-1914) Count TotalProduceRequestRate and TotalFetchRequestRate in BrokerTopicMetrics
[ https://issues.apache.org/jira/browse/KAFKA-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326763#comment-14326763 ] Aditya A Auradkar commented on KAFKA-1914: -- Created reviewboard https://reviews.apache.org/r/31168/diff/ against branch origin/trunk Count TotalProduceRequestRate and TotalFetchRequestRate in BrokerTopicMetrics - Key: KAFKA-1914 URL: https://issues.apache.org/jira/browse/KAFKA-1914 Project: Kafka Issue Type: Sub-task Components: core Reporter: Aditya A Auradkar Assignee: Aditya Auradkar Attachments: KAFKA-1914.patch, KAFKA-1914.patch, KAFKA-1914_2015-02-17_15:46:27.patch Currently the BrokerTopicMetrics only counts the failedProduceRequestRate and the failedFetchRequestRate. We should add 2 metrics to count the overall produce/fetch request rates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1729) add doc for Kafka-based offset management in 0.8.2
[ https://issues.apache.org/jira/browse/KAFKA-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326764#comment-14326764 ] Joel Koshy commented on KAFKA-1729: --- Created reviewboard https://reviews.apache.org/r/31169/diff/ against branch origin/0.8.2 add doc for Kafka-based offset management in 0.8.2 -- Key: KAFKA-1729 URL: https://issues.apache.org/jira/browse/KAFKA-1729 Project: Kafka Issue Type: Sub-task Reporter: Jun Rao Assignee: Joel Koshy Fix For: 0.8.2.0 Attachments: KAFKA-1729.patch, KAFKA-1729.patch, KAFKA-1782-doc-v1.patch, KAFKA-1782-doc-v2.patch, KAFKA-1782-doc-v3.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1914) Count TotalProduceRequestRate and TotalFetchRequestRate in BrokerTopicMetrics
[ https://issues.apache.org/jira/browse/KAFKA-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-1914: - Attachment: KAFKA-1914.patch Count TotalProduceRequestRate and TotalFetchRequestRate in BrokerTopicMetrics - Key: KAFKA-1914 URL: https://issues.apache.org/jira/browse/KAFKA-1914 Project: Kafka Issue Type: Sub-task Components: core Reporter: Aditya A Auradkar Assignee: Aditya Auradkar Attachments: KAFKA-1914.patch, KAFKA-1914.patch, KAFKA-1914_2015-02-17_15:46:27.patch Currently the BrokerTopicMetrics only counts the failedProduceRequestRate and the failedFetchRequestRate. We should add 2 metrics to count the overall produce/fetch request rates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31168: Patch for KAFKA-1914
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31168/#review73035 --- Ship it! Ship It! - Joel Koshy On Feb. 19, 2015, 12:01 a.m., Aditya Auradkar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31168/ --- (Updated Feb. 19, 2015, 12:01 a.m.) Review request for kafka. Bugs: KAFKA-1914 https://issues.apache.org/jira/browse/KAFKA-1914 Repository: kafka Description --- Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into trunk Fixing failing unit test Fixing failing unit test Diffs - core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 525c835803b048c952667567cd205b4b06824391 Diff: https://reviews.apache.org/r/31168/diff/ Testing --- All unit tests now pass. BUILD SUCCESSFUL Total time: 9 mins 35.296 secs Thanks, Aditya Auradkar
Re: Review Request 30809: Patch for KAFKA-1888
On Feb. 18, 2015, 12:06 a.m., Mayuresh Gharat wrote: core/src/main/scala/kafka/tools/ContinuousValidationTest.java, line 207 https://reviews.apache.org/r/30809/diff/1/?file=859055#file859055line207 This might end up in infinite loop if something goes wrong with cluster, right? Should we have a maximum numnber of retries? What do you think? This will not be an issue since for timed runs we will interrupt the thread anyway after a fixed time. This is the mode which is being used in the upgrade test. On Feb. 18, 2015, 12:06 a.m., Mayuresh Gharat wrote: core/src/main/scala/kafka/tools/ContinuousValidationTest.java, line 424 https://reviews.apache.org/r/30809/diff/1/?file=859055#file859055line424 Are you assuming that first argument will be some key? If you take a look at the script I am expecting alternate parameters like -timedRun -timeToSpawn Key is essentially the parameter name. On Feb. 18, 2015, 12:06 a.m., Mayuresh Gharat wrote: core/src/main/scala/kafka/tools/ContinuousValidationTest.java, line 474 https://reviews.apache.org/r/30809/diff/1/?file=859055#file859055line474 what do you mean by rebuild state later? What I meant was that between two runs for the rolling upgrade test we will not re-use any state from zookeeper or the brokers so I do not need to worry about clean shutdown. On Feb. 18, 2015, 12:06 a.m., Mayuresh Gharat wrote: core/src/main/scala/kafka/tools/ContinuousValidationTest.java, line 77 https://reviews.apache.org/r/30809/diff/1/?file=859055#file859055line77 Why we need a flip? The flip is needed to reset the get pointer in byte buffer to beginning of the byte buffer else we will get underflow exception. - Abhishek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/#review72786 --- On Feb. 18, 2015, 1:59 a.m., Abhishek Nigam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/ --- (Updated Feb. 18, 2015, 1:59 a.m.) Review request for kafka. Bugs: KAFKA-1888 https://issues.apache.org/jira/browse/KAFKA-1888 Repository: kafka Description --- patch for KAFKA-1888 Diffs - build.gradle 0f0fe60a74542efa91a0e727146e896edcaa38af core/src/main/scala/kafka/tools/ContinuousValidationTest.java PRE-CREATION system_test/broker_upgrade/bin/kafka-run-class.sh PRE-CREATION system_test/broker_upgrade/bin/test.sh PRE-CREATION system_test/broker_upgrade/configs/server1.properties PRE-CREATION system_test/broker_upgrade/configs/server2.properties PRE-CREATION system_test/broker_upgrade/configs/zookeeper_source.properties PRE-CREATION Diff: https://reviews.apache.org/r/30809/diff/ Testing --- Scripted it to run 20 times without any failures. Command-line: broker-upgrade/bin/test.sh dir1 dir2 Thanks, Abhishek Nigam
[jira] [Updated] (KAFKA-1729) add doc for Kafka-based offset management in 0.8.2
[ https://issues.apache.org/jira/browse/KAFKA-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1729: -- Attachment: KAFKA-1729_2015-02-18_17:30:37.patch add doc for Kafka-based offset management in 0.8.2 -- Key: KAFKA-1729 URL: https://issues.apache.org/jira/browse/KAFKA-1729 Project: Kafka Issue Type: Sub-task Reporter: Jun Rao Assignee: Joel Koshy Fix For: 0.8.2.0 Attachments: KAFKA-1729.patch, KAFKA-1729.patch, KAFKA-1729_2015-02-18_17:30:37.patch, KAFKA-1782-doc-v1.patch, KAFKA-1782-doc-v2.patch, KAFKA-1782-doc-v3.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31169: Patch for KAFKA-1729
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31169/ --- (Updated Feb. 19, 2015, 1:30 a.m.) Review request for kafka. Bugs: KAFKA-1729 https://issues.apache.org/jira/browse/KAFKA-1729 Repository: kafka Description (updated) --- Allow constructing explicitly versioned offset fetch request in javaapi Diffs (updated) - core/src/main/scala/kafka/javaapi/OffsetFetchRequest.scala 1c25aa3332f9e4f0222db715b524d9179b5306cf Diff: https://reviews.apache.org/r/31169/diff/ Testing --- Thanks, Joel Koshy
Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer
There have been a couple of rounds of this. Basically a bunch of complaints people have about the producer boil down to their being no limit on how long a request will block if the kafka cluster goes hard down. Some of the discussion was here, I think: https://issues.apache.org/jira/browse/KAFKA-1788 But a lot was on previous producer-related tickets. E.g. close() blocking forever if the kafka cluster is down happens because the requests never fail they just queue indefinitely waiting for kafka to come back. In any case for the purpose of this KIP we don't need to pick a mechanism or configuration for controlling client request timeout. All we are saying now is that we should add such a mechanism and when we do it will address any concerns about flush() blocking for an inderminate amount of time (so we don't need any kind of timeout on flush itself now). -Jay On Wed, Feb 18, 2015 at 4:24 PM, Joel Koshy jjkosh...@gmail.com wrote: Actually, could you clarify this a bit (since I'm not sure which thread you are referring to) - specifically, how would this tie in with the current timeout we have for the producer (for example)? On Tue, Feb 17, 2015 at 02:55:44PM -0800, Jay Kreps wrote: Yeah there was a separate thread on adding a client-side timeout to requests. We should have this in the new java clients, it just isn't there yet. When we do this the flush() call will implicitly have the same timeout as the requests (since they will complete or fail by then). I think this makes flush(timeout) and potentially close(timeout) both unnecessary. -Jay On Tue, Feb 17, 2015 at 2:44 PM, Guozhang Wang wangg...@gmail.com wrote: In the scala clients we have the socket.timeout config as we are using blocking IOs, when such timeout is reached the TimeoutException will be thrown from the socket and the client can handle it accordingly; in the java clients we are switching to non-blocking IOs and hence we will not have the socket timeout any more. I agree that we could add this client request timeout back in the java clients, in addition to allowing client / server's non-blocking selector to close idle sockets. Guozhang On Tue, Feb 17, 2015 at 1:55 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: I'm thinking the flush call timeout will naturally be the timeout for a produce request, No? Currently it seems we don¹t have a timeout for client requests, should we have one? ‹Jiangjie (Becket) Qin On 2/16/15, 8:19 PM, Jay Kreps jay.kr...@gmail.com wrote: Yes, I think we all agree it would be good to add a client-side request timeout. That would effectively imply a flush timeout as well since any requests that couldn't complete in that time would be errors and hence completed in the definition we gave. -Jay On Mon, Feb 16, 2015 at 7:57 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi All, Thanks Jay and all address concern. I am fine with just having flush() method as long as it covers failure mode and resiliency. e.g We had situation where entire Kafka cluster brokers were reachable, but upon adding new kafka node and admin migrated leader to new brokers that new brokers is NOT reachable from producer stand point due to fire wall but metadata would continue to elect new broker as leader for that partition. All I am asking is either you will have to give-up sending to this broker or do something in this scenario. As for the current code 0.8.2 release, caller thread of flush() or close() method would be blocked for ever so all I am asking is https://issues.apache.org/jira/browse/KAFKA-1659 https://issues.apache.org/jira/browse/KAFKA-1660 Also, I recall that there is timeout also added to batch to indicate how long message can retain in memory before expiring. Given, all this should this API be consistent with others up coming patches for addressing similar problem(s). Otherwise, what we have done is spawn a thread for just calling close() or flush with timeout for join on caller end. Anyway, I just wanted to give you issues with existing API and if you guys think this is fine then, I am ok with this approach. It is just that caller will have to do bit more work. Thanks, Bhavesh On Thursday, February 12, 2015, Joel Koshy jjkosh...@gmail.com wrote: Yes that is a counter-example. I'm okay either way on whether we should have just flush() or have a timeout. Bhavesh, does Jay's explanation a few replies prior address your concern? If so, shall we consider this closed? On Tue, Feb 10, 2015 at 01:36:23PM -0800, Jay Kreps wrote: Yeah we could do that, I guess I just feel
Re: [VOTE] KIP-8: Add a flush() method to the new producer
+1. On Wed, Feb 18, 2015 at 3:49 PM, Jay Kreps j...@confluent.io wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+method+to+the+producer+API +1 binding -Jay -- -- Guozhang
