[GitHub] kafka pull request #3468: KAFKA-5542: Improve Java doc for LeaderEpochFileCa...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/3468 KAFKA-5542: Improve Java doc for LeaderEpochFileCache.endOffsetFor() You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-5542 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3468.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3468 commit 88246b6e3c0142aa8f56c74280e07c5cb6cf99b2 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-06-30T13:00:54Z KAFKA-5542 Improve Java doc for LeaderEpochFileCache.endOffsetFor() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2831: MINOR: KAFKA-5036 (points 2, 5): Refactor caching ...
GitHub user benstopford opened a pull request:
https://github.com/apache/kafka/pull/2831
MINOR: KAFKA-5036 (points 2, 5): Refactor caching of Latest Epoch
This PR covers point (2) and point (5) from KAFKA-5036:
2. Currently, we update the leader epoch in epochCache after log append in
the follower but before log append in the leader. It would be more consistent
to always do this after log append. This also avoids issues related to failure
in log append.
5. The constructor of LeaderEpochFileCache has the following:
lock synchronized { ListBuffer(checkpoint.read(): _*) }
But everywhere else uses a read or write lock. We should use consistent
locking.
This is a refactor to the way epochs are cached, replacing the code to
cache the latest epoch in the LeaderEpochFileCache by reusing the cached value
in Partition. There is no functional change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/benstopford/kafka KAFKA-5036-part2-second-try
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2831.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2831
commit 3e9c130672824070968173b2991a43eb9fa139b6
Author: Ben Stopford <benstopf...@gmail.com>
Date: 2017-04-10T12:56:48Z
KAFKA-5036: Refactor the caching of the latest epoch. Workflow is simpler
if we resuse the value cached in partition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] kafka pull request #2821: MINOR: CollectionUtils.groupDataByTopic in Offsets...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2821 MINOR: CollectionUtils.groupDataByTopic in OffsetsForLeaderEpochRequest/Response You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka kip-101-cleanup-group-by Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2821.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2821 commit 59db990b56d5810fa941e596a1d98f9e7263c547 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-04-07T09:31:32Z MINOR: KIP-101 refactor requests/response to CollectionUtils.groupDataByTopic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2743: KIP-101: Alter Replication Protocol to use Leader ...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/2743 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2808: KIP-101: Alter Replication Protocol to use Leader ...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2808 KIP-101: Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation This PR describes the addition of Partition Level Leader Epochs to messages in Kafka as a mechanism for fixing some known issues in the replication protocol. Full details can be found here: [KIP-101 Reference](https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation) *The key elements are*: - Epochs are stamped on messages as they enter the leader. - Epochs are tracked in both leader and follower in a new checkpoint file. - A new API allows followers to retrieve the leader's latest offset for a particular epoch. - The logic for truncating the log, when a replica becomes a follower, has been moved from Partition into the ReplicaFetcherThread - When partitions are added to the ReplicaFetcherThread they are added in an initialising state. Initialising partitions request leader epochs and then truncate their logs appropriately. This test provides a good overview of the workflow `EpochDrivenReplicationProtocolAcceptanceTest.shouldFollowLeaderEpochBasicWorkflow()` The corrupted log use case is covered by the test `EpochDrivenReplicationProtocolAcceptanceTest.offsetsShouldNotGoBackwards()` Remaining work: The test `EpochDrivenReplicationProtocolAcceptanceTest.shouldSurviveFastLeaderChange()` doesn't correctly reproduce the underlying issue. This will be altered later to properly support this use case. You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka kip-101-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2808.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2808 commit a96a8bbee2435bd46cd19746f61b73eeb2f94088 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-03-27T16:16:16Z All work to date squashed (18 committs) KIP-101: Push after merge. KIP-101: Fixes for checksytle breaks KIP-101: Remove TestSuite class KIP-101: Comments KIP-101: Comments KIP-101: Altered logic in ReplicaFetcherThread: - On NoLeaderForPartition continue to poll for epochs indefinitely - Add synchronisation around log trucation to ensure we cannot truncate the log of a leader (light testing, more to follow, noted in TODO) KIP-101: Rename Epoch -> PartitionLeaderEpoch KIP-101: First commit based on feedback from Jun/Jason KIP-101: Second commit based on feedback from Jun/Jason KIP-101: Third commit based on feedback from Jun/Jason KIP-101: Fourth commit based on feedback from Jun/Jason - removed retainMatchingOffset parameter from clearOldest as not used KIP-101: tidy only KIP-101: Return Log End Offset If Undefined Epoch Requested (this covers the case of a bootstrapping broker) KIP-101: Altered log truncation to always be inclusive, so we always delete epochs inclusive of the passed offset, whether clearing earliest or latest entries. KIP-101: Add optimisation back in for previous commit. KIP-101: If epochOffset.endOffset() is UNSUPPORTED_EPOCH_OFFSET, which can happen during the transition phase, we should fall back to HW. Improved fuglyness too. KIP-101: Small tidy KIP-101: Refactored threading model in Abstract/ReplicaFetcherThread. Functionally identical but now the logic sits largely in the abstract class. KIP-101: Moved OffsetsForLeaderEpoch.getResponseFor() into ReplicaManager KIP-101: (1) Altered ReplicaFetcherThread to poll continuously on errors. (2) Only send epoch requests if version >= 11 KIP-101: As segments are recovered, truncate the epoch cache with the appropriate segment KIP-101: Fix bug in DummyFetcherThread which was defaulting to requiring initialisation. Caused AbstractFetherThread test to hang. KIP-101: Fix bug in ReplicaManager imports KIP-101: Fix bug in ReplicaManager imports by making all imports explicit. Also remove OffsetCheckpointFile which appears to still be in the remote repostiory. This was causing a compilation issue. KIP-101: Remove override of OffsetsTopicPartitionsProp (to 5) in PlaintexConsumerTest as it causes a test in BaseConsumerTest to fail. Will fix this issue in separate PR KIP-101: Rename only (OffsesForLeaderEpochRequest) KIP-101: Fix merge error KIP-101: Fix couple more merge errors KIP-101: Re-enable test_zk_security_upgrade on Ismael's request
[GitHub] kafka pull request #2743: KIP-101: Alter Replication Protocol to use Leader ...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2743 KIP-101: Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation This PR describes the addition of Partition Level Leader Epochs to messages in Kafka as a mechanism for fixing some known issues in the replication protocol. Full details can be found here: [KIP-101 Reference](https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation) *The key elements are*: - Epochs are stamped on messages as they enter the leader. - Epochs are tracked in both leader and follower in a new checkpoint file. - A new API allows followers to retrieve the leader's latest offset for a particular epoch. - The logic for truncating the log, when a replica becomes a follower, has been moved from Partition into the ReplicaFetcherThread - When partitions are added to the ReplicaFetcherThread they are added in an initialising state. Initialising partitions request leader epochs and then truncate their logs appropriately. This test provides a good overview of the workflow `EpochDrivenReplicationProtocolAcceptanceTest.shouldFollowLeaderEpochBasicWorkflow()` The corrupted log use case is covered by the test `EpochDrivenReplicationProtocolAcceptanceTest.offsetsShouldNotGoBackwards()` You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka kip-101-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2743.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2743 commit 768dfeb9c5ef8b78b48fd898311d2cb459e564fd Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-03-27T16:16:16Z KIP-101: Push after merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2734: KAFKA-4948: Failure in kafka.admin.DescribeConsume...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2734 KAFKA-4948: Failure in kafka.admin.DescribeConsumerGroupTest... This test fails regularly when run on OSX (on my machine anyway). The fix decreases the number of partitions in the consumer offsets topic allows it to complete within the timeout. As an aside I wondered why it was taking so long (regression??). This is why: - It takes 6 seconds to create the offsets topic (on OSX) - It takes ~200ms to create each log - It takes ~ 100ms to create each index file (time index + offset index) - This time is spent on the line [raf.setLength()](https://github.com/apache/kafka/blob/5fc530bc483db145e0cba3b63a57d6d6a7c547f2/core/src/main/scala/kafka/log/AbstractIndex.scala#L56) This isn't an issue in practice, but at least we know this isn't a real regression. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka quick-fix-desc-cons-grp-test-KAFKA-4948 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2734.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2734 commit 95f0deaafe354d932dc31d0494116a9391a61897 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-03-24T17:49:25Z KIP-101: Merge from dev branch into this branch brining in all changes over EOS commit 4f96770d4f735090ea045d39404c9d44ff7f4e01 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-03-24T19:11:27Z KAFKA-4948: 5 partitions should suffice. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2698: MINOR: Increase Throttle lower bound assertion in ...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2698 MINOR: Increase Throttle lower bound assertion in throttling_test You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka throttling-test-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2698.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2698 commit 693435b8770f7c25dc498fb8e74776530bbba329 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-03-16T21:16:08Z MINOR: increase thottle bound in test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2399: KAFKA 4596: Throttled Replica Reassignment Error
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2399 KAFKA 4596: Throttled Replica Reassignment Error Fixes a logic error in the Reassignment process which throws an exception if you don't rebalance all partitions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4596 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2399.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2399 commit a731d18f404af815e2c7ee20585ef15ff3b9af22 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-17T20:25:24Z KAFKA-4596: add support for moving a subset of partitions with throttle engaged commit 3e523a544c9d32637672507c8533864d2c8046fe Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T11:21:28Z KAFKA-4596: comment commit a7018d9a26d665c83a7585a0cb2eaa4dc9426739 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T14:08:56Z KAFKA-4596: Added richer functinoal test for reassigment. commit 863c94fe12f086d0df925127bfe26f7693391212 Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T17:03:36Z KAFKA-4596: Cleaned up logic commit 37d2da4e578ae2852cae8037a8ab7e2da607e30f Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T17:06:38Z KAFKA-4596: tidy commit bba5492cf5ce05ae1d71290e9c1a255ba0813cfd Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T17:09:54Z KAFKA-4596: revert imports commit 3c136fc70a582748cd6e5deabfcba237de3f1adc Author: Ben Stopford <benstopf...@gmail.com> Date: 2017-01-18T17:13:47Z KAFKA-4596: whitespace --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2034: KAFKA-4309: Allow "pluggable" properties in KafkaS...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2034 KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka throttling-system-test-kafka-changes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2034.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2034 commit 81704e9aac6583b29a615f4095b7ee0e66dec06d Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-17T21:25:39Z KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2033: Documentation for Throttled Replication
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/2033 Documentation for Throttled Replication You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka throttling-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2033.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2033 commit becce0b29a3db3d70ecf1ddc45138d3403088de4 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-17T20:58:39Z KAFKA-4266 - formatting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1997: KAFKA-4266: ReassignPartitionsClusterTest - Ensure...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1997 KAFKA-4266: ReassignPartitionsClusterTest - Ensure ZK publication completed before start Increase the reliability of the one temporal comparison in ReassignPartitionsClusterTest by imposing a delay after ZK is updated. This should be more reliable than just increasing the amount of data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4266 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1997.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1997 commit 010b0b6a1a23d6297c62417aa4165b2c89142e1d Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-07T10:01:16Z KAFKA-4266 - check if each broker has had the throttled replicas updated before proceeding with the test (where applicable) commit bcc9e1ab81ecd5537a64a15271a52712514d5820 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-08T11:57:00Z KAFKA-4266 - revert prev change as it's not really useful commit 9a243dbfe81d2ccc99a153d9693fafe535f15b07 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-08T12:03:16Z KAFKA-4266 - add delay after ZK update for throttle config. commit 7e9f0e90d3d8adfbf05635c3c3e6c16c0d9f5c00 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-10-08T12:16:59Z KAFKA-4266 - formatting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1932: KAFKA-4225: Replication Quotas: Control Leader & F...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1932 KAFKA-4225: Replication Quotas: Control Leader & Follower Limit Separately You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4225-over-KAFKA-4216 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1932 commit e4fe714e8b695c08963ecd830fad60f4a47a7110 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-29T08:26:05Z KAFKA-4225: Replication Quotas: Control Leader & Follower Limit Separately --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1928: KAFKA-4225: Replication Quotas: Control Leader & F...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/1928 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1928: KAFKA-4225: Replication Quotas: Control Leader & F...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1928 KAFKA-4225: Replication Quotas: Control Leader & Follower Limit Separately See final commit. (depends on https://github.com/apache/kafka/pull/1906) You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4225-over-KAFKA-4216 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1928.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1928 commit ddcb115fa6a098cef79209ae8022a4467b8ea873 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-26T14:26:11Z KAFKA-4216: First cut of split of leader/follwer replica lists. commit bee043a85f4f8034785f3f83447e732dcd8d316e Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-26T15:34:41Z KAFKA-4216: Refactored the way we calculate which replicas should be throttled. No functional change in this commit. commit 1c416f465ccfe6c9c57c2e6e86c71616034e8b99 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T09:12:05Z KAFKA-4177: Refactored the way we derive the throttled replica list (it was super ugly). No functional change commit 637316e23a4cd3290101dd62080158a36c63f57c Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T09:17:54Z KAFKA-4177: Formatting only based on Jun's feedback commit 2e20c50c94242e9a1a59cfdb826cb68739e5bed0 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T09:24:03Z KAFKA-4177: remove reformatting of license file text commit 70f263e1b6371bb27a1afd33299cf378cd8d3f67 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T14:36:46Z KAFKA-4216: small formatting change. commit bc07a348283aa307bde6a32d0cf7e84cb7b1ad64 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T14:46:52Z KAFKA-4216: Improved comments commit bab1c38a370f8d0b73ac9a3ed40880da4ee95e38 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T10:59:46Z KAFKA-4216: Consolidated code that creates properties commit 97acd19854b19564b9db4aac3166d18fdcd9ce0f Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T14:59:47Z KAFKA-4216: Change throttled replica list property to be of type LIST not STRING commit 17af1e728d3e32a44ea0cbcee1967e31ce401ca1 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T15:26:38Z KAFKA-4216: Stylistic / logging changes only commit faedb50f8646a24da94bac2ed047dcda326bdb8d Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:13:07Z KAFKA-4216: Added validation to ensure that all partitions in the proposed assignment exist in the cluster commit b39a4a4870dca46750b4063e79b1b8fc8aaf79df Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:23:13Z KAFKA-4216: Small change to add distinct to the topic selection commit ae65db17eb53e5822e7bc137fc94c2ee7c73392f Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:24:42Z KAFKA-4216: Alteration to previous commit. One of the distict calls was not needed. commit dd2d161cb0dc5ad785f93427cfc7a63c9ebbdb14 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:26:34Z KAFKA-4216: Removed long line commit 5080f9560b7657e552564aca35e633a322631b48 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:34:39Z KAFKA-4216: Ismael's suggested '(proposed(tp).toSet -- current)' change commit 67ac7bea374764100a79033e4245f7c09a6d435b Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:37:07Z KAFKA-4216: Ismael's suggestion to format method commit 797fded9d9b9d1df1f4b76fbbae74dad07b98ee8 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:40:13Z KAFKA-4216: Stylistic changes. commit 5d06a265847d558ad60c1b82c9bfb1b30dce47b0 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T16:41:11Z KAFKA-4216: Add text to fail() commit 84b5602e01636bb8ffa7cdabe1e9b782d67f5e22 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T17:32:58Z KAFKA-4216: silly error in test. commit 784c5cdf81be1abf13b8959cf7d3f3bbef2c2d12 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-28T19:40:52Z KAFKA-4225: Replication Quotas: Control Leader & Follower Limit Separately --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1906: KAFKA 4216: Control Leader & Follower Throttled Re...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1906 KAFKA 4216: Control Leader & Follower Throttled Replicas Separately Splits the throttled replica configuration (the list of which replicas should be throttled for each topic) into two. One for the leader throttle, one for the follower throttle. So: quota.replication.throttled.replicas => quota.leader.replication.throttled.replicas & quota.follower.replication.throttled.replicas You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4216-seperate-leader-and-follower-throttled-replica-lists Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1906.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1906 commit 03a786d4cf1bf73e7c096c226d8e3984e73d5c6f Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-26T14:26:11Z KAFKA-4216: First cut of split of leader/follwer replica lists. commit 1570292da52a656da608669be57b7759a248af87 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-26T15:34:41Z KAFKA-4216: Refactored the way we calculate which replicas should be throttled. No functional change in this commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1896: KAFKA-4200: Fix throttle argument in kafka-reassig...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1896 KAFKA-4200: Fix throttle argument in kafka-reassign-partitions.sh Simple jira which alters two things: 1. kafka-reassign-partitions --verify prints Throttle was removed regardless of whether a throttle was applied. It should only print this if the value was actually changed. 2. --verify should exception if the âthrottle argument. (check generate too) To test this I extracted all validation logic into a separate method and added a test which covers the majority of combinations. The validation logic was retained as is, other than implementing (2) and adding validation to the --broker-list option which you can currently apply to any of hte main actions (where it is ignored). Requirement 1 was tested manually (as it's just println). Testing: - Build passes locally. - System test reassign_partitions_test.py also passes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4200 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1896 commit f85efead9c468d5d6e906a5c11f573a36d620244 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-21T09:26:50Z KAFKA-4200: kafka-reassign-partitions â-verify prints "Throttle was removed" regardless of whether a throttle was applied. It should only print this if the value was actually changed. Tested this maually ensuring the value is not printed when we call verify whilst the rebalance is running, but does print the value when running verify when the throttle completes. Rerunning verify after the throttle completes also does not print the line. commit 843c00c3cbee354fcfb1f0fdda5f77addd4b6eb4 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-21T12:15:37Z KAFKA-4200: kafka-reassign-partitions â-verify prints "Throttle was removed" regardless of whether a throttle was applied. It should only print this if the value was actually changed. Tested this maually ensuring the value is not printed when we call verify whilst the rebalance is running, but does print the value when running verify when the throttle completes. Rerunning verify after the throttle completes also does not print the line. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1893: KAFKA-4178: Windows in Rate Calculation
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1893 KAFKA-4178: Windows in Rate Calculation ReplicationQuotas included a new Rate class which changed the way the window is calculated. @junrao asked that we look to consolidate this. On balance I believe there is a good case for both, so this PR pulls out two policies for calculating the window, and attempts to explain what they are and why we might need them. They are: **Elapsed**: Set the window to the difference between the oldest and newest measurement time. Handle NaN. _(Replication Quotas use this)_ **Fixed**: Fix the window to the full duration (10s by default). _(Client Quotas use this)_ Replication Quotas uses **Elapsed** as **Fixed** provides underestimates during the first window. Underestimates are best avoided as they create a load spike when replication starts. So it's preferable to overestimate the rate, and hence increase the throttled rate slowly. Elapsed is also significantly easier to test. I'm totally sure why Client Quotas were changed to use **Fixed**. I know there was a NaN issue, but should really be tangential (it is handled in both policies). However it seems sensible to slowly throttle on a client's connection down to the desired rate when you initialise or change quotas. So I think there is a requirement for both types of rate, and hence I've included both in this PR. **Elapsed** suits throttled replication, where the general use case is to initiate some rebalance and immediately apply a conservative throttle. **Fixed** suits client quotas where we want to gently throttle a client down to the desired rate over a period of seconds. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4178 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1893 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1892: KAFKA-[TBD]: Make ReassignPartitionsTest System Te...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1892 KAFKA-[TBD]: Make ReassignPartitionsTest System Test move data The ReassignPartitionsTest system tests doesn't reassign any replicas (i.e. move data). This is a simple issue. It uses a 3 node cluster with replication factor of 3, so whilst the replicas are jumbled around, nothing actually is moved from machine to machine when the assignment is executed. This fix just ups the number of nodes to 4 so things move. Tests pass locally. There are runs pending on the two branch builders (#94, #551) No Jira included as Jira is currently down. Will update. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka fix_reassignment_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1892.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1892 commit 0327f1bc70ce37e1f82d519d2a3a95de0062e45a Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-20T12:39:00Z KAFKA-TBD: Tiny, but important, fix to reassignment test, which currently doesn't reassign any data. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1881: KAFKA-4193: Fix for Intermittent failure in Fetche...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1881 KAFKA-4193: Fix for Intermittent failure in FetcherTest You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4193 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1881.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1881 commit 573bdd41cf954875e23cd53fb65f7ac56dc19345 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-19T17:03:38Z KAFKA-4193: Fix for intermittant failure in FetcherTest caused by erroneously altered acks setting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1873: KAFKA-4184: Intermitant failures in ReplicationQuo...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1873 KAFKA-4184: Intermitant failures in ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4184 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1873.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1873 commit 812edfebf27a942ea26c1e0c74010e233932aad2 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-17T09:32:10Z KAFKA-4184: Because we don't have long to run the test, replication can be uneven across brokers. Instead take the average rate and increase the tollerance a bit commit 2ea94074254bf10a8ad862e742112a3fa5e7d3d1 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-17T10:12:04Z KAFKA-4184: Split out rate comparison into bonds checks rather than equality --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1864: Kafka-4177: Remove ThrottledReplicationRateLimit f...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1864 Kafka-4177: Remove ThrottledReplicationRateLimit from Server Config This small PR pulls ThrottledReplicationRateLimit out of KafkaConfig and puts it in a class that defines Dynamic Configs. Client configs are also placed in this class and validation added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-4177 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1864.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1864 commit 63fbfe2f464b92aba7513c90086feed5d3afca33 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-16T14:26:01Z KAFKA-4177: Minor refactor to pull the dynamic broker property replication.quota.throttled.rate out of KafkaConfig and its own class collocated with the ConfigCommand/AdminUtils commit 72ddadd42088127d2d4bdfb812079145e534bb1e Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-16T14:43:26Z KAFKA-4177: Refactored to include dynamic client configs also commit d51be065112ed398b82c966a38ad8f2ef8fc0f04 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-16T15:08:16Z KAFKA-4177: Move to kafka.server package commit 495c17c53a0c64e8ea539dad52508a20ca7b4c5c Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-16T15:13:14Z KAFKA-4177: Moved tests to its own class commit 3a609d9f52c0be49c4b37055f3e73f6002b2e31d Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-16T15:16:02Z KAFKA-4177: Whitespace only --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1776: KIP-73 - Replication Quotas
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1776 KIP-73 - Replication Quotas This applies to Replication Quotas based on KIP-73 [()](https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas) originally motivated by KAFKA-1464. **This first PR demonstrates the approach**. Main stuff left to do: - The quota value should be moved to a dynamic, broker-level config. Here we are making do with a client-level override. - The fetch size is currently unbounded. This will be addressed in KIP-74, but we need to ensure this ensures requests donât go beyond the throttle window. - There is no specific validation or other utilities in ConfigCommand or code in ReassignPartitionsCommand for validation etc. - Tests are largely happy path, the functional tests rely on a single broker-to-broker set up. There are no system tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka rep-quotas-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1776.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1776 commit d4964b8905cef3d1e11b20a3ed8964643e214ad3 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-02T22:44:06Z CPKAFKA-291: Port of original POC, which used seperate replica fetcher threads. commit a3d1a5590df491468a42a481faaa89c7951114d4 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-10T15:13:34Z CPKAFKA-296: MINOR: Final checkin .. includes failing test demonstrating the current linearisibility issue commit 300ecc43db4bda3adc5ee47e505324a4b16b15df Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-11T15:59:35Z CPKAFKA-305: POC of leader, bytes based throttling. This commit includes the rollback of the previous work. commit 3fc0444e9d77c65449c08a6c7ac482da09ebc431 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-11T16:23:08Z CPKAFKA-305: MINOR: rename and tidy only commit 0c542440d41d1924a4c8d16f692cd814bcb1a460 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-11T16:23:33Z CPKAFKA-305: MINOR: rename and tidy only commit 627905b4f4083dcc7ec46b1e4d25d50952f294ad Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-11T18:25:00Z CPKAFKA-305: Completed POC of follower throttling. Quick and Dirty. commit 5f4c6aeb30f538b23e8aa53d5783386491648319 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-12T10:25:18Z CPKAFKA-305: Added test shouldReplicateThrottledAndNonThrottledPartitionsConcurrently commit 239d6901d45518b448c90ccf9b59f5d9d3481361 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-15T16:39:03Z CPKAFKA-305: Stabalised tests by replacing Rate with SimpleRate. This is likely a temporary measure, but that depends on whether we want a graduated start as defined in the current rate implementation. Altered purgatory check. Currently not using code for proposing rate changes, instead we validate simple whether the quota is currently exceeded (boolean) commit 86425e44d90194ef96f3d666035b9b0977f7e510 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-15T18:52:16Z CPKAFKA-305: Added small test around replication quota manager commit e26c8621ae6c550b901c2f9533960675f03cde22 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-16T07:10:14Z CPKAFKA-305: Altered leader algorithm to evaluate bytes per partition commit 434b1cc3e0326f8dbc6c153b7435b1f83e661747 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-16T15:17:57Z CPKAFKA-305: SReplicaManagerQuotasTest - added support for multiple (subsequent) partitions commit a6cc2400defaec16308fd3ccfc05410069d42c0b Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-16T15:36:59Z CPKAFKA-305: Moved things around in ClientQuotaManager, fixed bug in ReplicationQuotaTest commit 2d3ab68b57c205775b8a065ed09e35919aa90820 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-16T16:55:27Z CPKAFKA-305: Added a second partition to one of the acceptance tests. Fixed bug in logging in KafkaApis commit ab3d846a50a6589d4b860c8afb2bda695a2547e7 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-17T16:05:59Z CPKAFKA-305: small changes, nothing functional commit b8a2fcd71a792f7ae06a7aa2993141f14d0554e4 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-17T20:44:12Z CPKAFKA-305: Extracted ReplicationQuotaManager into its own class, and associated changes. commit 5cd27799b4e09ece2d11c22de9f6e13f09ad5c11 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-08-22T12:26:06Z CPKAFKA-305: Moved kafka
[GitHub] kafka pull request: MINOR: Documentation for Rack Awareness
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/1369 MINOR: Documentation for Rack Awareness You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka rack-awareness-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1369.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1369 commit 8600abfdeaaca40eb08749671e9fc2ae8abbf310 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-05-11T14:21:28Z Added documentation for rack awareness commit 5021cfff1efe41a62fd1a6b273fb30d568441750 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-05-11T14:23:47Z removed newline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: Kafka 3270 Added some Happy Path Tests for the...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/956 Kafka 3270 Added some Happy Path Tests for the Reassign Partitions Command You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-3270-ReassignPartitionsCommand-Tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/956.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #956 commit 1851c214265c8bc58fa5d6539ac14b85ebc8dd55 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-02-23T20:06:50Z KAFKA-3270: Happy path tests for replica reassignment commit eac1957c716e416b1c80952fb588095ced3d6c9c Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-02-23T20:55:37Z KAFKA-3242: tidy commit 1d36c7f1ee9e63873342ae11e46c5fd0d46a6d90 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-02-23T21:00:35Z KAFKA-3242: Oops - small delete of log line --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3242: minor rename / logging change to C...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/924 KAFKA-3242: minor rename / logging change to Controller KAFKA-3242: minor rename / logging change to references to 'adding partitions' to indicate 'modifying partitions' You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka small_changes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #924 commit e2dfcf98b70dfd3a97b30cad25b6b74abbd191b1 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-02-16T22:30:49Z KAFKA-3242: minor rename / logging change to refererences to 'adding partitions' to indicate 'modifying partitions' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2964: Split Security Rolling Upgrade Tes...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/667 KAFKA-2964: Split Security Rolling Upgrade Test by Client and Broker Protocols The core of this test is to ensure we evaluate enabling security in a running cluster where we have different broker and client protocols. Also in this PR are some improvements to the validation process in produce_consume_validate.py: - Fail fast if producer or consumer stop running. - If messages go missing, check in the data files to see if the cause was data loss or the consumer missing messages. - Remove unnecessary sleeps which hide problem with consumer disconnection. - Make it possible for the ConsoleConsumer to optionally log both what it consumed and when it consumed it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka security-rolling_upgrade-additions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #667 commit c0ae7f29b8381d83870bb56d4312a4897c77944e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-08T21:27:01Z KAFKA-2964: Parameterise security rolling upgrade so it runs independetly for broker-broker and broker-client protocols. commit 03740f9fd21c05eb8eb3492ac3ce35efa1f9428a Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-10T19:38:44Z KAFKA-2964: Check for data loss if messages go missing commit 9640d3efaeb0a0ef549b3a8c5a758bfde60ace30 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-10T22:34:31Z KAFKA-2964: Refactored produce_consume_validate commit bd53895892d7938f17975298f1380739a2f45504 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-11T09:21:12Z KAFKA-2964: clean up --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: Add Rolling Upgrade Notes to Security Docs
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/625 Add Rolling Upgrade Notes to Security Docs And added info about the krb5.conf file as we don't appear to mention that in the current docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka security_docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/625.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #625 commit 5a0058ea528899cd5bb5b2b38e3c82b3fd0c23d8 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-03T15:14:45Z Include krb5 parameter in SASL docs commit 8b4b6f277f8130ce3d1286c89da748b9d2a8a46c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-03T15:47:36Z Added documentation for rolling upgrade commit 46ffa30bbe16629e583275bbfd25c8e3acf4da8e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-03T16:00:44Z formatting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2915: Fix problem with System Tests that...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/608 KAFKA-2915: Fix problem with System Tests that use bootstrap.servers embedded in jinja files Fixes problems in mirror maker and consumer tests http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/290/ http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/289/ You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2915-jinja-bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/608.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #608 commit 640532b7ca10298d545e523e291e6f6fe82843c6 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-01T15:27:46Z KAFKA-2915: Added call security protocol to bootstrap servers call in jinja file commit 192d96c6a53481db5b8dc428f0a2eb6d401862ea Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-12-01T16:40:56Z KAFKA-2915: fixed string formatting --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2899: (trivial) Log unexpected exception...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/593 KAFKA-2899: (trivial) Log unexpected exceptions thrown when reading local log Currently we don't log exceptions raised when reading from the local log which makes tracking down the cause of problems a bit tricky. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka small-patches Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/593.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #593 commit c30313871fb539ba4d484145fda7b94e88b89921 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-26T18:52:55Z KAFKA-2899: Should log unexpected exceptions thrown when reading from local log --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2824: MiniKDC based tests don't run in V...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/520 KAFKA-2824: MiniKDC based tests don't run in VirtualBox This is a hack which works. Is there a better way? Build of the replication_test.py running here: http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/174/console You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka fix-for-sasl-virtual-box Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #520 commit e398ae0af9020463fb3750d384fe00beb05c9ab2 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T22:21:23Z KAFKA-2824: force open binding in minikdc commit 57fe6425b72cd0a977f3f8c9c46239f2feab9fef Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T23:54:46Z KAFKA-2824: add back in clean up commit b1f2943e2ff10327120c06907b20f6f46510f8b7 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T23:56:58Z KAFKA-2824: tidy import --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2824: MiniKDC based tests don't run in V...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/520 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2791: removed deprecated properties
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/525 KAFKA-2791: removed deprecated properties Removed support for BLOCK_ON_BUFFER_FULL_CONFIG (block.on.buffer.full) Removed support for METADATA_FETCH_TIMEOUT_CONFIG Removed support for TIMEOUT_CONFIG (aka timeout.ms) Added support for MAX_BLOCK_MS_CONFIG Added support for REQUEST_TIMEOUT_MS_CONFIG You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2791 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #525 commit 0452dccf43b9f096d3f28ea65b18fcc96acf23cc Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-13T17:25:20Z KAFKA-2791: removed deprecated properties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2824: MiniKDC based tests don't run in V...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/520 KAFKA-2824: MiniKDC based tests don't run in VirtualBox This is a hack which works. Is there a better way? Build of the replication_test.py running here: http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/174/console You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka fix-for-sasl-virtual-box Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #520 commit e398ae0af9020463fb3750d384fe00beb05c9ab2 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T22:21:23Z KAFKA-2824: force open binding in minikdc commit 57fe6425b72cd0a977f3f8c9c46239f2feab9fef Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T23:54:46Z KAFKA-2824: add back in clean up commit b1f2943e2ff10327120c06907b20f6f46510f8b7 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-12T23:56:58Z KAFKA-2824: tidy import --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2771: Added rolling upgrade system test ...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/496 KAFKA-2771: Added rolling upgrade system test (docktape) for SSL This still needs a final run of the full system test suite. I've run this test and the replication_test.py (which is the other test most affected) on Ec2. Both pass. Not totally happy with the logic around listener selection but the SASL tests moved over a model with a single port which doesn't work well when performing a rolling bounce. To make this stable on Ec2 I increased the consumer timeouts and use a unique consumer group. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka security-upgrade-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/496.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #496 commit ee34dbeb0667f081f77ac32031b571724ecd819b Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-11-09T00:09:17Z KAFKA-2771: Added rolling upgrade system test (docktape) for SSL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2338: add force option to topic / config...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/351 KAFKA-2338: add force option to topic / config command so they can be called programatically Tiny change to add a force option to the topic and config commands so they can be called programatically without requiring user input. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka CPKAFKA-61B Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/351.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #351 commit b29316e7174f21a91043d7a3aa451b6345324dd2 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-10-22T12:27:13Z KAFKA-2338: add 'force' option to avoid console prompts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2338: Warn on max.message.bytes change
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/322 KAFKA-2338: Warn on max.message.bytes change - Both TopicCommand and ConfigCommand warn if message.max.bytes increases - Log failures on the broker if replication gets stuck due to an oversized message You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka CPKAFKA-61 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/322.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #322 commit d59110d28fe1dcd62520e2b031005a369ceb4bc7 Author: benstopford <benstopf...@gmail.com> Date: 2015-10-15T17:13:09Z KAFKA-2338: Warn user if they override max.message.bytes. Warn on failing replica fetches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2637: Cipher suite setting should be con...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/301 KAFKA-2637: Cipher suite setting should be configurable for SSL Enables Cipher suite setting. Code was previously reviewed by @ijuma, @harshach. Moving to an independent PR. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka cipher-switch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/301.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #301 commit 1ad5d1b631651d51328bd939c3856843536002c6 Author: benstopford <benstopf...@gmail.com> Date: 2015-10-13T09:13:54Z KAFKA-2637: Cipher suite setting should be configurable for SSL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2638: Added default properties file to C...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/302 KAFKA-2638: Added default properties file to ConsumerPerformance Blocker for SSL integration You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2638 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #302 commit f44ea477455adf72985da47c0e48fbd392a00331 Author: benstopford <benstopf...@gmail.com> Date: 2015-10-13T11:44:01Z KAFKA-2638: Added default properties file via consumer.config command line option --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2638: Added default properties file to C...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2431: Easier Testing of SSL
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/217 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2431: Easier Testing of SSL
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/217 KAFKA-2431: Easier Testing of SSL - Allow cipher suites to be specified relevant properties - Avoid System.exit in ProducerPerformance so this can be externally invoked - Add command line option so that a default properties can be specified in ConsumerPerformance (needed for ssl properties) You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka ssl-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #217 commit 18fb3f0914dc284c5632522065d7e5be062ed62a Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-08T00:36:51Z KAFKA-2431: easier testing of SSL - Avoid System.exit in ProducerPerformance so this can be externally invoked - Add command line option so that a default set of properties can be speicfied in ConsumerPerformance (needed for ssl properties) - Allow cipher suites to be specified vi properties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 71b11fac33a67ef9f0ac8ac09bcbb5305a56047f Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 43d6a0678fd37d7f382d5f89c898571ae4e3cfbb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. commit cac85029c7d802da29d68073545c118804bd41cb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T17:08:29Z KAFKA-2453: Added additional arguments to call to EndToEndLatency from Performance tests commit 119a6fa545bcaf586c9cb110f0e13c1cfee1f56c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T10:20:14Z KAFKA-2453: Rebased to trunk KAFKA-2453: removed whitespace KAFKA-2453: Formatting only commit 954f076701dd8961e3f835b08a022fc31ae74943 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T15:30:17Z KAFKA-2453: Incorporate changes from KAFKA-2486 Previous version used an optional busy loop to get better performance by avoiding sleeps inside the API. These turned out to be a bug fixed in KAFKA-2486 so the optional busy loop has been removed. commit 4b96fab3037e6ed2ee00a0ff370168000fddcd09 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T18:17:29Z KAFKA-2453: removed sleep which I believe is not longer needed now we have consumer.seekToEnd() commit 3e52bca1f0f711b7abdee680253f808bb871057e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T22:04:17Z KAFKA-2453: Producer acks can be a string commit 29ce7cd9c95980f391bf3315099d285c120ace6e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T13:13:06Z KAFKA-2453: Feedback from Gwen + fix to seek problem - Fixed issue with seekToEnd evaluating lazily (i.e. when poll is called) meaning messages can be missed in slower environments (discovered when I ran this on EC2). Detailed in comments. - Removed redundant retry backoff override (this was an artifact of KAFKA-2486) - Forced producer acks to be 1 or all (i.e. synchronous) - Reduced poll to a reasonable timeout to avoid hangs in erroneous situations - Added check for results being non zero - Added check that there is only a single message returned commit d2d7378c7fdaf0a095141ece1dff8c52aa72a9ac Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T14:09:33Z KAFKA-2453: downgrade duplicate message exception to warning commit a6544d19cb8d1da58a67f878d8204c10a3c42c1a Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T16:05:12Z KAFKA-2453: Added in support for ssl properties file. Ismael's changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/158 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 71b11fac33a67ef9f0ac8ac09bcbb5305a56047f Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 43d6a0678fd37d7f382d5f89c898571ae4e3cfbb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. commit cac85029c7d802da29d68073545c118804bd41cb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T17:08:29Z KAFKA-2453: Added additional arguments to call to EndToEndLatency from Performance tests commit 119a6fa545bcaf586c9cb110f0e13c1cfee1f56c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T10:20:14Z KAFKA-2453: Rebased to trunk KAFKA-2453: removed whitespace KAFKA-2453: Formatting only commit 954f076701dd8961e3f835b08a022fc31ae74943 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T15:30:17Z KAFKA-2453: Incorporate changes from KAFKA-2486 Previous version used an optional busy loop to get better performance by avoiding sleeps inside the API. These turned out to be a bug fixed in KAFKA-2486 so the optional busy loop has been removed. commit 4b96fab3037e6ed2ee00a0ff370168000fddcd09 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T18:17:29Z KAFKA-2453: removed sleep which I believe is not longer needed now we have consumer.seekToEnd() commit 3e52bca1f0f711b7abdee680253f808bb871057e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T22:04:17Z KAFKA-2453: Producer acks can be a string commit 29ce7cd9c95980f391bf3315099d285c120ace6e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T13:13:06Z KAFKA-2453: Feedback from Gwen + fix to seek problem - Fixed issue with seekToEnd evaluating lazily (i.e. when poll is called) meaning messages can be missed in slower environments (discovered when I ran this on EC2). Detailed in comments. - Removed redundant retry backoff override (this was an artifact of KAFKA-2486) - Forced producer acks to be 1 or all (i.e. synchronous) - Reduced poll to a reasonable timeout to avoid hangs in erroneous situations - Added check for results being non zero - Added check that there is only a single message returned --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/158 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 71b11fac33a67ef9f0ac8ac09bcbb5305a56047f Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 43d6a0678fd37d7f382d5f89c898571ae4e3cfbb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. commit cac85029c7d802da29d68073545c118804bd41cb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T17:08:29Z KAFKA-2453: Added additional arguments to call to EndToEndLatency from Performance tests commit 119a6fa545bcaf586c9cb110f0e13c1cfee1f56c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T10:20:14Z KAFKA-2453: Rebased to trunk KAFKA-2453: removed whitespace KAFKA-2453: Formatting only commit 954f076701dd8961e3f835b08a022fc31ae74943 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T15:30:17Z KAFKA-2453: Incorporate changes from KAFKA-2486 Previous version used an optional busy loop to get better performance by avoiding sleeps inside the API. These turned out to be a bug fixed in KAFKA-2486 so the optional busy loop has been removed. commit 4b96fab3037e6ed2ee00a0ff370168000fddcd09 Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T18:17:29Z KAFKA-2453: removed sleep which I believe is not longer needed now we have consumer.seekToEnd() commit 3e52bca1f0f711b7abdee680253f808bb871057e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T22:04:17Z KAFKA-2453: Producer acks can be a string commit 29ce7cd9c95980f391bf3315099d285c120ace6e Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T13:13:06Z KAFKA-2453: Feedback from Gwen + fix to seek problem - Fixed issue with seekToEnd evaluating lazily (i.e. when poll is called) meaning messages can be missed in slower environments (discovered when I ran this on EC2). Detailed in comments. - Removed redundant retry backoff override (this was an artifact of KAFKA-2486) - Forced producer acks to be 1 or all (i.e. synchronous) - Reduced poll to a reasonable timeout to avoid hangs in erroneous situations - Added check for results being non zero - Added check that there is only a single message returned commit d2d7378c7fdaf0a095141ece1dff8c52aa72a9ac Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-02T14:09:33Z KAFKA-2453: downgrade duplicate message exception to warning --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 71b11fac33a67ef9f0ac8ac09bcbb5305a56047f Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 43d6a0678fd37d7f382d5f89c898571ae4e3cfbb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. commit cac85029c7d802da29d68073545c118804bd41cb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T17:08:29Z KAFKA-2453: Added additional arguments to call to EndToEndLatency from Performance tests commit 119a6fa545bcaf586c9cb110f0e13c1cfee1f56c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T10:20:14Z KAFKA-2453: Rebased to trunk KAFKA-2453: removed whitespace KAFKA-2453: Formatting only --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/158 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford reopened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 71b11fac33a67ef9f0ac8ac09bcbb5305a56047f Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 43d6a0678fd37d7f382d5f89c898571ae4e3cfbb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. commit cac85029c7d802da29d68073545c118804bd41cb Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-08-21T17:08:29Z KAFKA-2453: Added additional arguments to call to EndToEndLatency from Performance tests commit 119a6fa545bcaf586c9cb110f0e13c1cfee1f56c Author: Ben Stopford <benstopf...@gmail.com> Date: 2015-09-01T10:20:14Z KAFKA-2453: Rebased to trunk KAFKA-2453: removed whitespace KAFKA-2453: Formatting only --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/158 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2453: Enable new consumer in EndToEndLat...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/158 KAFKA-2453: Enable new consumer in EndToEndLatency You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2453b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit 9088400f4a1cf0eea4a94e2ce8773ccdb23edb21 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-21T16:44:48Z KAFKA-2453: migrated EndToEndLatencyTest to new consumer API. Added feature for configuring message size. Added inline assertion. commit 614102350c1553086f7fcedb3db42e1114e0f4d7 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-21T16:45:19Z KAFKA-2453: small change which prevents the ConsoleConsumer from throwing an exception when the Finalizer thread tries to close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: Kafka 2015 - Enable ConsoleConsumer to use new...
GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/144 Kafka 2015 - Enable ConsoleConsumer to use new consumer This extends the original patch done by GZ to provide Console access to both the new and old consumer API's. The code follows a pattern similar to that already used in ConsoleProducer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka KAFKA-2015 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/144.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #144 commit 68dc02dcdff89ed20c6b5fab9d24b618ad25c714 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T12:36:13Z KAFKA-2015 commit 6735a6af16d25fa256f8cf91890694316d1e6950 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T12:41:43Z KAFKA-2015 commit 6e23964d608a19509aa9880d8d2944ad795bacd6 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T12:42:17Z KAFKA-2015 commit 68ce8d789a98705e3ee82fbb6c7727db6c8434e2 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T17:00:57Z KAFKA-2015 - implemented base case keeping the existing BaseConsumer design (which in truth I'm not super-fond of but it retains consistency with BaseProducer. This checkin still includes TopicPartition, rather than Topic subscription. commit dc540f25d0b4708e93c79cbbadf851bc83b8fe5c Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T17:29:42Z KAFKA-2015 - changed to use topic subscription rather than topicpartition subscription commit b793f0e9ce0e6496f6fee8322668c2d8149ef573 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T18:01:46Z KAFKA-2015 - tidied leftovers from original Jira Patch from GZ commit cb5681d27d387a7c428ead3d962bef8acf6b5ff0 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-13T18:15:17Z KAFKA-2015 - fix indentation commit 3072e5825e2715ae28cd48fb2bda2b51468d2373 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-14T15:42:39Z KAFKA-2015 - added some limited testing commit ba25f099adcc50ef8b5d77c8fe5c1ba137ae02fd Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-14T18:18:14Z KAFKA-2015 - couple more tests commit 3d0638f5da2e0d35c74ad6081ac12786103d1d92 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-14T18:20:20Z KAFKA-2015 - couple more tests commit 9d01e8ad292039abcc8201de74467b4b4b8af601 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-14T18:41:47Z KAFKA-2015 - added formatters back in commit cb0ede65b5c343b6396992a147fc10510e65dbdf Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-14T18:52:09Z KAFKA-2015 - formatting only commit 2e9ade93a88641bcdbf1e736dd9d4874a47e854b Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-15T18:03:39Z KAFKA-2015 - added back in previous change to remove randomly created group ids from ZK in the shutdown hook commit de06323b7b75aa4450df16e5315c0db70de06fe0 Author: Ben Stopford benstopf...@gmail.com Date: 2015-08-17T07:53:02Z KAFKA-2015 - whitespace only --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
