[GitHub] kafka pull request #4146: MINOR: Tighten up locking when aborting expired tr...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4146 MINOR: Tighten up locking when aborting expired transactions This is a followup to #4137 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-followups-to-bump-epoch-on-expire-patch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4146.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4146 commit 1c8cc6672f93315f1027fe7cf1dae9975cf7871e Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-27T16:55:33Z Tighten up locking and address other minor comments ---
[GitHub] kafka pull request #4137: KAFKA-6119: Bump epoch when expiring transactions ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4137 KAFKA-6119: Bump epoch when expiring transactions in the TransactionCoordinator A description of the problem is in the JIRA. I have added an integration test which reproduces the original scenario, and also added unit test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-6119-bump-epoch-when-expiring-transactions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4137 commit 4405e4f9c30e82864417fdbafe3817ee4acee661 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-26T00:58:44Z Bump the epoch when we abort a transaction on the coordinator commit 9945b4f8315dc8c82aaeb003c07458a6231ee96c Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-26T01:06:04Z Lock the transaction metadata before fencing the epoch. Make the case matching exhaustive ---
[GitHub] kafka pull request #4057: KAFKA-6053: Fix NoSuchMethodError when creating Pr...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4057 KAFKA-6053: Fix NoSuchMethodError when creating ProducerRecords with older client versions You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-6053-fix-no-such-method-error-in-producer-record Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4057.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4057 commit 221a96da1dc6221dd3f61786d5fc1119b6848a7f Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-11T16:58:44Z Fix NoSuchMethodError when creating ProducerRecords with older client versions ---
[GitHub] kafka pull request #3266: WIP: KAFKA-5403: Transaction system test consumer ...
Github user apurvam closed the pull request at: https://github.com/apache/kafka/pull/3266 ---
[GitHub] kafka pull request #4020: KAFKA-6003: Accept appends on replicas and when re...
Github user apurvam closed the pull request at: https://github.com/apache/kafka/pull/4020 ---
[GitHub] kafka pull request #4039: MINOR: Bump the request timeout for the transactio...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4039 MINOR: Bump the request timeout for the transactional message copier Multiple inflights means that when there are rolling bounces and other cluster instability, there is an increased likelihood of having previously tried batch expire in th accumulator. This is a fatal error for a transaction, causing the copier to exit. To work around this, we bump the request timeout. We can get rid of this when KIP-91 is merged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-bump-request-timeout-in-transactional-message-copier Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4039 commit 5ad017fd90b75b13b6683efaac17454d9cc5bb26 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-07T17:50:33Z Bump the reuest timeout for the transactional message copier to fix the transient system test failures ---
[GitHub] kafka pull request #4029: KAFKA-6016: Make the reassign partitions system te...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4029 KAFKA-6016: Make the reassign partitions system test use the idempotent producer With these changes, we are ensuring that the partitions being reassigned are from non-zero offsets. We also ensure that every message in the log has producerId and sequence number. This means that it successfully reproduces https://issues.apache.org/jira/browse/KAFKA-6003, as can be seen below: ``` [2017-10-05 20:57:00,466] ERROR [ReplicaFetcher replicaId=1, leaderId=4, fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread) kafka.common.KafkaException: Error processing data for partition test_topic-16 offset 682 at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:171) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:168) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:168) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:168) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:168) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:218) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:166) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:109) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) Caused by: org.apache.kafka.common.errors.UnknownProducerIdException: Found no record of producerId=1000 on the broker. It is possible that the last message with the producerId=1000 has been removed due to hitting the retention limit. ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-6016-add-idempotent-producer-to-reassign-partitions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4029.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4029 commit af48d74be4f2c4473d8f97664ff0f3e450bfe3ec Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-05T05:27:23Z Initial commit trying to create the scenario where we are creating a replica from scratch but starting from a non zero sequence when doing so. commit 9566f91b00a5a7c249823107e4792b844809ccca Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-05T05:52:24Z Use retention bytes to force segment deletion commit 6087b3ed01472d24677623c9b3ef92a3678da96f Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-05T21:16:47Z Configure the log so that we can reproduce the case where we are building producer state from a non zero sequence ---
[GitHub] kafka pull request #4020: KAFKA-6003: Accept appends on replicas and when re...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4020 KAFKA-6003: Accept appends on replicas and when rebuilding the log unconditionally This is a port of #4004 for the 0.11.0 branch. With this patch so that we _only_ validate appends which originate from the client. In general, once the append is validated and written to the leader the first time, revalidating it is undesirable since we can't do anything if validation fails, and also because it is hard to maintain the correct assumptions during validation, leading to spurious validation failures. For example, when we have compacted topics, it is possible for batches to be compacted on the follower but not on the leader. This case would also lead to an OutOfOrderSequencException during replication. The same applies to when we rebuild state from compacted topics: we would get gaps in the sequence numbers, causing the OutOfOrderSequence. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAKFA-6003-0.11.0-handle-unknown-producer-on-replica Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4020.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4020 commit 0a6a0213c091c8e6b6a9c5ce7655b7e0d06c9db0 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-04T20:42:17Z KAFKA-6003: Accept appends on replicas and when rebuilding state from the log unconditionally. With this patch so that we _only_ validate appends which originate from the client. In general, once the append is validated and written to the leader the first time, revalidating it is undesirable since we can't do anything if validation fails, and also because it is hard to maintain the correct assumptions during validation, leading to spurious validation failures. For example, when we have compacted topics, it is possible for batches to be compacted on the follower but not on the leader. This case would also lead to an OutOfOrderSequencException during replication. The same applies to when we rebuild state from compacted topics: we would get gaps in the sequence numbers, causing the OutOfOrderSequence. ---
[GitHub] kafka pull request #4004: KAFKA-6003: Accept appends on replicas uncondition...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/4004 KAFKA-6003: Accept appends on replicas unconditionally when local producer state doesn't exist Without this patch, if the replica's log was somehow truncated before the leader's it is possible for the replica fetcher thread to continuously through an OutOfOrderSequenceException because the incoming sequence would be non-zero and there is no local state. This patch changes the behavior so that the replica state is updated to the leader's state if there was no local state for the producer at the time of the append. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-6003-handle-unknown-producer-on-replica Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4004.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4004 commit 341a0d2ba3ec0716f1830860869bf773f1bf8d85 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-10-02T22:41:19Z KAFKA-6003: Accept appends on replicas unconditionally when local producer state doesn't exist. Without this patch, if the replica's log was somehow truncated before the leader's it is possible for the replica fetcher thread to continuously through an OutOfOrderSequenceException because the incoming sequence would be non-zero and there is no local state. This patch changes the behavior so that the replica state is updated to the leader's state if there was no local state for the producer at the time of the append. ---
[GitHub] kafka pull request #3969: KAFKA-5888: System test to check ordering of messa...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3969 KAFKA-5888: System test to check ordering of messages with transactions and max.in.flight > 1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5888-system-test-which-check-ordering Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3969.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3969 commit bb6b3cf2545984a03d8177d6eaa205261540a0dd Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-27T01:18:27Z Initial commit of transaction system test which checks for message ordering commit 2dc508603266407f1fc5de10954a8807bfa85440 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-27T04:16:56Z Small refactor to use the ducktape @matrix feature to run tests which check order as well as tests that don't ---
[GitHub] kafka pull request #3947: KAFKA-5959: Fix NPE in Sender.canRetry when idempo...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3947 KAFKA-5959: Fix NPE in Sender.canRetry when idempotence is not enabled You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5959-npe-in-sender Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3947.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3947 commit 29a4522500f720ad3ccb1e37cd2133d449986c81 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-22T18:59:51Z Fix NPE in when idempotence is not enabled ---
[GitHub] kafka pull request #3896: KAFKA-5914 add message format version and message ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3896 KAFKA-5914 add message format version and message max bytes to metadata response Updated the `TopicResponse` part of the `MetadataResponse` to include the message format version and the message max bytes. One problem here is that we use the `TopicConfigHandler` to listen for topic changes. However this is not invoked for topic _creates_ since the change notification path is not updated during creates. I am not sure what the right solution is here. Intuitively, we should update the the change notification path for topic creates, but not sure if that has compatibility (or other) implications. TODO: 1. Add a more complete integration test where the client sends a real `MetadataRequest` and receives the proper `MetadataResponse`. 2. Rebase to incorporate Jason's changes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5914-add-message-format-version-and-message-max-bytes-to-metadata-response Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3896 commit 7cc943c30be8bef4646580f19e5191ef7e476b98 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-19T04:29:00Z Initial commit with a few tests commit 5099d5163b071020cc627b6b0a7c4f388de99eaa Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-19T06:05:43Z Added one more test ---
[GitHub] kafka pull request #3878: KAFKA-5913: Add the RecordMetadataNotAvailableExce...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3878 KAFKA-5913: Add the RecordMetadataNotAvailableException We return this exception from `RecordMetadata.offset()` or `RecordMetadata.timestamp()` if these pieces of metadata were not returned by the broker. This will happen, for instance, when the broker returns a `DuplicateSequenceException`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5913-add-record-metadata-not-available-exception Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3878.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3878 commit e9770a65203f7bdcea8c25a5fbaaa9366d12851c Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-16T01:15:15Z Initial commit ---
[GitHub] kafka pull request #3877: MINOR: Disable KafkaAdminClientTest.testHandleTime...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3877 MINOR: Disable KafkaAdminClientTest.testHandleTimeout This test is super flaky in the PR builder. https://issues.apache.org/jira/browse/KAFKA-5792 tracks the fix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-disable-adminclient-timeout-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3877.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3877 commit d533e0c598fa5a96591ff46c20df29897596d250 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-15T20:09:51Z Disable KafkaAdminClientTest.testHandleTimeout ---
[GitHub] kafka pull request #3865: KAFKA-5793: Tighten up the semantics of the OutOfO...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3865 KAFKA-5793: Tighten up the semantics of the OutOfOrderSequenceException *WIP : Don't review yet, still to add tests* Description of the solution can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Solving+the+problem+of+spurious+OutOfOrderSequence+errors You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5793-tighten-up-out-of-order-sequence-v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3865.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3865 commit fb44876987bd2f75c900b187fdc755da3f85114f Author: Apurva Mehta <apu...@confluent.io> Date: 2017-09-15T01:01:58Z Initial commit of the client and server code, with minimal tests ---
[GitHub] kafka pull request #3743: KAFKA-5494: enable idempotence with max.in.flight....
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3743 KAFKA-5494: enable idempotence with max.in.flight.requests.per.connection > 1 Here we introduce client and broker changes to support multiple inflight requests while still guaranteeing idempotence. Two major problems to be solved: 1. Sequence number management on the client when there are request failures. When a batch fails, future inflight batches will also fail with `OutOfOrderSequenceException`. This must be handled on the client with intelligent sequence reassignment. We must also deal with the fatal failure of some batch: the future batches must get different sequence numbers when the come back. 2. On the broker, when we have multiple inflights, we can get duplicates of multiple old batches. With this patch, we retain the record metadata for 5 older batches. I have added `TODO(reviewers)` comments for specific decisions in the code which are worth discussing. TODO: 1. Add more unit tests, especially for loading different snapshot versions correctly, more client side unit tests, more broker side tests to validate that we are caching the correct number of batches (some of this is already there). 2. Update the system tests to check for ordering. 3. Run a tight loop of system tests. 4. Add comments about the assumptions made around the network client semantics of send/receive. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5494-increase-max-in-flight-for-idempotent-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3743.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3743 commit 005eee527ab425d8e3d8678aad4b5305cde6ca08 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-12T00:25:06Z Initial commit of client side changes with some tests commit 63bf074a38ec3efef728863081805a36d9111038 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-17T23:49:10Z Implemented broker side changes to cache extra metadata. Todo: 1) Write more unit tests. 2) Handle deletion / retention / cleaning correctly. commit 1ad49f30f03ff665f5657680cbcc5e045210ce45 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-23T00:10:39Z Change the client side code so that the sequence numbers are assigned and incremented during drain. If a batch is retried, it's sequence number is unset during the completion handler. If the first inflight batch returns an error, the next sequence to assign is reset to the last ack'd sequence + 1. commit d9b86b7cb8e7001a7d5fc42a2ec061ebd0332a6a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-24T01:33:54Z WIP commit 9ff885fe6db7172d28ea8fe406972a7763c0a49d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-25T06:23:50Z Implemented log cleaning functionality with tests commit 5508a194c74a8946a8451c01814324e6ba788cfe Author: Apurva Mehta <apu...@confluent.io> Date: 2017-08-25T19:27:03Z Fix merge issues aftre rebasing onto trunk --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3550: KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3550 KAFKA-5610: KafkaApis.HandleWriteTxnMarkerRequest should return UNKNOWN_TOPIC_OR_PARTITION on partition emigration. Before this patch, we would return the non-retriable `UNSUPPORTED_FOR_MESSAGE_FORMAT` error when leadership changed, causing markers to be lost. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5610-handleWriteTxnMarker-should-handle-emigration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3550.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3550 commit 3ceba90e31964dc13e264376c993aadfe29e632c Author: Apurva Mehta <apu...@confluent.io> Date: 2017-07-19T22:10:33Z Return UKNOWN_TOPIC_OR_PARTITION when partition emigrates during KafkaApis.handleWriteTxnMarkerRequest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3462: KAFKA-5543: Remove all partition metrics when a to...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3462 KAFKA-5543: Remove all partition metrics when a topic is deleted You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5543-remove-lastStableOffsetLag-metric Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3462.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3462 commit f315bbdcd0a26755db1a7169822279f8965c5309 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-29T23:41:58Z Remove all partition metrics when a topic is deleted --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3441: MINOR: Enable the TransactionsBounceTest
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3441 MINOR: Enable the TransactionsBounceTest I'll let this have multiple runs on the branch builder to see if it fails, and investigate if so. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-enable-transactions-bounce-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3441.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3441 commit e9286e72cb061245f400a6b32e20f245084522cb Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-27T05:22:56Z Enable the TransactionsBounceTest to see if it is stable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3400: Enable transactions in ProducerPerformance Tool
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3400 Enable transactions in ProducerPerformance Tool With this patch, the `ProducePerfomance` tool can create transactions of differing durations. This patch was used to to collect the initial set of benchmarks for transaction performance, documented here: https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit#gid=282787170 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-add-transaction-size-to-producre-perf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3400.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3400 commit a0ae1f00d84dc3516646366b63e59915796f95a5 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-20T00:11:21Z Enable the performance producer to cerate transactions of variable durations. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3377: KAFKA-5477: Lower retryBackoff for AddPartitionsRe...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3377 KAFKA-5477: Lower retryBackoff for AddPartitionsRequest This patch lowers the retry backoff when receiving a CONCURRENT_TRANSACTIONS error from an AddPartitions request. The default of 100ms would mean that back to back transactions would be 100ms long at minimum, making things to slow. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka HOTFIX-lower-retry-for-add-partitions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3377.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3377 commit 0d676688e7ed9a8d63189eb704143e62752707cc Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-20T00:36:28Z Lower retryBackoff when receiving a CONCURRENT_TRANSACTIONS error from an AddPartitions request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3374: KAFKA-5032: Update the docs for message size confi...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3374 KAFKA-5032: Update the docs for message size configs across the board Before 0.11, we used to have limits for maximum message size on the producer, broker, and consumer side. From 0.11 onward, these limits apply to record batches as a whole. This patch updates the documentation of the configs to make this explicit. A separate patch will have more extensive upgrade notes to tie all the changes together in one narrative. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5032-message-size-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3374 commit b8e1379a54d21141a22694b2aa6d422709bfb89f Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-19T21:20:03Z Change references to 'message' in the size options to 'record batch', since everything is written and read in batches in the current version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3360: KAFKA-5020: Update message format in implementatio...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3360 KAFKA-5020: Update message format in implementation docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5020-message-format-docs-update Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3360.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3360 commit 35e7f55d5067aa503467eb84f5de12fb1ea33b3f Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-17T00:10:57Z Update message format in implementation docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3355: KAFKA-5457: MemoryRecordsBuilder.hasRoomFor should...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3355 KAFKA-5457: MemoryRecordsBuilder.hasRoomFor should account for record headers You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5457-memoryrecordsbuilder-has-room-for-should-account-for-headers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3355.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3355 commit 51ebbef40d3f98d4fbc5c041e53064f529853542 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-16T00:52:23Z Ensure that MemoryRecordsBuilder.hasRoomFor accounts for the headers in a record as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3353: KAFKA-5455 - Better Javadocs for the transactional...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3353 KAFKA-5455 - Better Javadocs for the transactional producer You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5455-proper-javadocs-eos-clients Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3353 commit e3e66ca85beba0a37f5d8af29be7413c3e15 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-15T21:27:51Z Initial commit of better Javadocs for the transactional producer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3346: MINOR: Cleanups for TransactionsTest
Github user apurvam closed the pull request at: https://github.com/apache/kafka/pull/3346 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3346: MINOR: Cleanups for TransactionsTest
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3346 MINOR: Cleanups for TransactionsTest The motivation is that KAFKA-5449 seems to indicate that producer instances can be shared across tests, and that producers from one test seem to be hitting brokers in another test. So this patch does two things: # Make transactionsTest use random ports in each test case. # Clear producers and consumers between tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-transactiontest-should-inherit-from-integration-test-harness Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3346.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3346 commit 2cc3aa4c77d80bd1c806f7ee276cb18ffbeaa540 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-15T00:35:52Z Make transactionsTest use random ports in each test case. Clear producers and consumers between tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3343: WIP: KAFKA-5449: fix bad state transition in trans...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3343 WIP: KAFKA-5449: fix bad state transition in transaction manager The `kafka.api.TransactionsTest.testReadCommittedConsumerShouldNotSeeUndecidedData` very rarely sees the following. I have run it 700 times locally without failure, so it only happens on jenkins. this PR adds trace logging to the client. Will keep running the PR builder here and hope that the test fails again so that we can understand what's going on. It is strange that we we have an ongoing send when we are in `READY` state. It is even more strange that we see a `ProducerFencedException` in the log. Could it be that some other run is interfering with this one (since multiple test cases use the same producer ids) ? ``` [2017-06-13 23:58:09,644] ERROR Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender:381) org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. [2017-06-13 23:58:10,177] ERROR [ReplicaFetcherThread-0-0]: Error for partition [topic2,3] to broker 0:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:10,177] ERROR [ReplicaFetcherThread-0-0]: Error for partition [topic2,0] to broker 0:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:10,178] ERROR [ReplicaFetcherThread-0-0]: Error for partition [topic1,2] to broker 0:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:12,128] ERROR ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes (org.apache.zookeeper.server.ZooKeeperServer:472) [2017-06-13 23:58:12,134] ERROR ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes (org.apache.zookeeper.server.ZooKeeperServer:472) [2017-06-13 23:58:12,310] ERROR [ReplicaFetcherThread-0-1]: Error for partition [topic1,0] to broker 1:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:12,311] ERROR [ReplicaFetcherThread-0-1]: Error for partition [topic1,3] to broker 1:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:15,998] ERROR ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes (org.apache.zookeeper.server.ZooKeeperServer:472) [2017-06-13 23:58:16,005] ERROR ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes (org.apache.zookeeper.server.ZooKeeperServer:472) [2017-06-13 23:58:16,177] ERROR [ReplicaFetcherThread-0-2]: Error for partition [topic1,2] to broker 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:16,177] ERROR [ReplicaFetcherThread-0-0]: Error for partition [topic1,3] to broker 0:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:16,178] ERROR [ReplicaFetcherThread-0-0]: Error for partition [topic1,0] to broker 0:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. (kafka.server.ReplicaFetcherThread:99) [2017-06-13 23:58:28,177] ERROR Uncaught error in kafka producer I/O thread: (org.apache.kafka.clients.producer.internals.Sender:164) org.apache.kafka.common.KafkaException: Invalid transition attempted from state READY to state ABORTABLE_ERROR at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:476) at org.apache.kafka.clients.producer.internals.TransactionManager.transitionToAbortableError(TransactionManager.java:289) at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:601) at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:272) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:223
[GitHub] kafka pull request #3313: KAFKA-5438: Fix UnsupportedOperationException in w...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3313 KAFKA-5438: Fix UnsupportedOperationException in writeTxnMarkersRequest Before this patch, the `partitionErrors` was an immutable map. As a result if a single producer had a marker for multiple partitions, and if there were multiple response callbacks for a single append, we would get an `UnsupportedOperationException` in the `writeTxnMarker` handler. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5438-fix-unsupportedoperationexception-in-writetxnmarker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3313.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3313 commit 0fc523aefb8a30f4fdebf166f904a780b40f6965 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-13T04:47:59Z Make partitionErrors a mutable map in KafkaApis.handleWriteTxnMarkerRequest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3308: KAFKA-5437: Always send a sig_kill when cleaning t...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3308 KAFKA-5437: Always send a sig_kill when cleaning the message copier When the message copier hangs (like when there is a bug in the client), it ignores the sigterm and doesn't shut down. this leaves the cluster in an unclean state causing future tests to fail. In this patch we always send SIGKILL when cleaning the node if the process isn't already dead. This is consistent with the other services. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5437-force-kill-message-copier-on-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3308.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3308 commit 5364c4830f5f97fbbfbefbf2c92e1d7230490ebf Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-12T22:52:45Z Always send a sig_kill when cleaning the message copier --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3286: KAFKA-5415: Remove timestamp check in completeTran...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3286 KAFKA-5415: Remove timestamp check in completeTransitionTo This assertion is hard to get right because the system time can roll backward on a host due to NTP (as shown in the ticket), and also because a transaction can start on one host and complete on another. Getting precise clock times across hosts is virtually impossible, and this check makes things fragile. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5415-avoid-timestamp-check-in-completeTransition Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3286.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3286 commit ccf5217d5a5985e7e88b2794c5fe43ff5b1d8a58 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-09T22:51:31Z Remove timestamp check in completeTransitionTo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3285: KAFKA-5422: Handle multiple transitions to ABORTAB...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3285 KAFKA-5422: Handle multiple transitions to ABORTABLE_ERROR correctly You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5422-allow-multiple-transitions-to-abortable-error Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3285.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3285 commit a3d0d923a76269d55541294447967167c35baebb Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-09T21:39:49Z Handle multiple transitions to ABORTABLE_ERROR correctly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3278: WIP: MINOR: Add logging and a small bug fix for th...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3278 WIP: MINOR: Add logging and a small bug fix for the transaction coordinator You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-add-logging-to-transaction-coordinator-in-all-failure-cases Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3278.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3278 commit 02b3816a626910ed9cdf08746c61de83d7c039aa Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-09T00:20:19Z Add logging and a small bug fix for the transaction coordinator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3266: KAFKA-5403: Transaction system test consumer shoul...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3266 KAFKA-5403: Transaction system test consumer should dedup messages by offset Since the consumer can consume duplicate offsets due to rebalances, we should dedup consumed messages by offset in order to ensure that the test doesn't fail spuriously. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5403-dedup-consumed-messages-transactions-system-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3266.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3266 commit f4ba943358aad08950e0952f60eb98843efd42f2 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-08T01:40:40Z WIP commit 5a6ac9dcc83c5bb0f62a5c387923612c5a9f1212 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-08T02:51:48Z WIP commit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3261: MINOR: Set log level for producer internals to tra...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3261 MINOR: Set log level for producer internals to trace for transactions test We need to debug most issues with the transactions system test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-set-log-level-for-producer-to-trace-for-transactions-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3261.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3261 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3252: KAFKA-5385: ProducerBatch expiry should go through...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3252 KAFKA-5385: ProducerBatch expiry should go through Sender.failBatch Before this patch, we would call `producerBatch.done` directly from the accumulator when expiring batches. This meant that we would not transition to the `ABORTABLE_ERROR` state in the transaction manager, allowing other transactional requests (including Commits!) to go through, even the produce failed. This patch modifies the logic so that we call `Sender.failBatch` on every expired batch, thus ensuring that the transaction state is accurate. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5385-fail-transaction-if-batches-expire Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3252.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3252 commit 20768261ad83c8ce13ab135d22907d3f35013e34 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-06T22:33:33Z WIP commit 3b50c5ed56cb696c708c59676f818f4bc0a3a3be Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-07T04:50:44Z Batch expiry should go through Sender.failBatch so that the transactional state is set correctly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3218: KAFKA-5373 : Revert breaking change to console con...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3218 KAFKA-5373 : Revert breaking change to console consumer This patch b63e41ea78a58bdea78be33f90bfcb61ce5988d3 since it broke the console consumer -- the consumer prints the addresses of the messages instead of the contents with that patch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5373-fix-console-consumer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3218 commit 245d0d83ddf1d190fefe24794182be29b0b0aaa8 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T23:33:54Z Revert b63e41ea78a58bdea78be33f90bfcb61ce5988d3 since it broke the console consumer -- the consumer prints the addresses of the messages instead of the contents with that patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3217: KAFKA-5366: Add concurrent reads to transactions s...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3217 KAFKA-5366: Add concurrent reads to transactions system test This currently fails in multiple ways. One of which is most likely KAFKA-5355, where the concurrent consumer reads duplicates. During broker bounces, the concurrent consumer misses messages completely. This is another bug. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5366-add-concurrent-reads-to-transactions-system-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3217 commit cd0990784aaa26fa6485e9f369a600b85c1647f9 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T22:25:00Z Add a concurrent consumer in the transactions system tests. This will exercise the abort index commit 71fcad197b403ff7873d646feec287d92793cbe6 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T22:29:47Z Bounce brokers as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3216: HOTFIX: Reinstate the placeholder for logPrefix in...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3216 HOTFIX: Reinstate the placeholder for logPrefix in TransactionManager You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka HOTFIX-logging-bug-in-transaction-manager Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3216.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3216 commit 975c6c673abfef9e70ca17c1b0a6f1efe4ada927 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T22:26:18Z Reinstate the placeholder for log prefix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3204: KAFKA-5322: Add an `OPERATION_NOT_ATTEMPTED` error...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3204 KAFKA-5322: Add an `OPERATION_NOT_ATTEMPTED` error code In the `AddPartitionsToTxn` request handling, if even one partition fails authorization checks, the entire request is essentially failed. However, the `AddPartitionsToTxnResponse` today will only contain the error codes for the topics which failed authorization. It will have no error code for the topics which succeeded, making it inconsistent with other APIs. This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is returned for the successful partitions to indicate that they were not added to the transaction. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5322-add-operation-not-attempted-for-add-partitions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3204.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3204 commit f454d495df20764105a42fe5e66351683e1a23ac Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T04:27:30Z WIP commit 764d8dc713ffde5703d58eb2a485567c6552f557 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T04:58:02Z Update comments and client code commit 7651210a7156a54376ff8a827b9220f60ff0ff20 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T05:38:49Z Add test, fix checkstyle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3202: KAFKA-5364: Don't fail producer if drained partiti...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3202 KAFKA-5364: Don't fail producer if drained partition is not yet in transaction Due to the async nature of the producer, it is possible to attempt to drain a messages whose partition hasn't been added to the transaction yet. Before this patch, we considered this a fatal error. However, it is only in error if the partition isn't in the queue to be sent to the coordinator. This patch updates the logic so that we only fail the producer if the partition would never be added to the transaction. If the partition of the batch is yet to be added, we will simply wait for the partition to be added to the transaction before sending the batch to the broker. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5364-ensure-partitions-added-to-txn-before-send Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3202.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3202 commit 7dd2f7f9c3c59c80ed0cac4965df819a8c7ddcbb Author: Apurva Mehta <apu...@confluent.io> Date: 2017-06-02T00:18:08Z Fix logic in ensurePartitionAddedToTransaction to account for partitions which would be added to the transaction. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3184: KAFKA-5351: Reset pending state when returning an ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3184 KAFKA-5351: Reset pending state when returning an error in appendTransactionToLog Without this patch, future client retries would get the `CONCURRENT_TRANSACTIONS` error code indefinitely, since the pending state wouldn't be cleared when the append to the log failed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5351-clear-pending-state-on-retriable-error Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3184 commit 518284b3a04b7df11c9689ac36a1fea8b50e852d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-31T22:07:24Z Reset pending state when returing an error in appendTransactionToLog --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3149: KAFKA-5281: System tests for transactions
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3149 KAFKA-5281: System tests for transactions You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5281-transactions-system-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3149 commit 07d38380e8734349666e0ddef0da469bfd2a38a4 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-25T00:58:20Z Initial commit of TransactionalMessageCopier program commit 9a711d7a0ff9c9a3ecb0aefe153a9eb411265609 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-25T21:17:18Z Initial commit of transactions system test commit 679df52ad77f731baca5eb5a74d1ad317856096a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-25T21:45:27Z WIP commit fc62578008353cac254ea76aa5eef8dbf2b35c9d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-26T05:13:05Z WIP - Got the basic copy / validate to work. Now to add bouncing and failures commit aa88be80623909d2bda2b2536db690e905fe702d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-26T06:10:50Z Implemented clean and hard bounces of brokers. Hard bounce test fails because we can't find the coordinator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3132: KAFKA-5147: Add missing synchronization to Transac...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3132 KAFKA-5147: Add missing synchronization to TransactionManager The basic idea is that exactly three collections, ie. `pendingRequests`, `newPartitionsToBeAddedToTransaction`, and `partitionsInTransaction` are accessed from the context of application threads. The first two are modified from the application threads, and the last is read from those threads. So to make the `TransactionManager` truly thread safe, we have to ensure that all accesses to these three members are done in a synchronized block. I inspected the code, and I believe this patch puts the synchronization in all the correct places. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5147-transaction-manager-synchronization-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3132.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3132 commit 9c869396d09712ebc76ef03c55507cd099b47914 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-24T05:28:15Z Add missing synchronization to TransactionManager --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3119: KAFKA-5273: Make KafkaConsumer.committed query the...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3119 KAFKA-5273: Make KafkaConsumer.committed query the server for all partitions Before this patch the consumer would return the cached offsets for partitions in its current assignment. This worked when all the offset commits went through the consumer. With KIP-98, offsets can be committed transactionally through the producer. This means that relying on cached positions in the consumer returns incorrect information: since commits go through the producer, the cache is never updated. Hence we need to update the `KafkaConsumer.committed` method to always lookup the server for the last committed offset to ensure it gets the correct information every time. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5273-kafkaconsumer-committed-should-always-hit-server Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3119.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3119 commit 17eb7eab70a40e3d4208a56463bb418350f80950 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-22T23:36:38Z Make KafkaConsumer.committed hit the server for all partitions, even those in its current assignment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3108: KAFKA-5247: Materialize committed offsets in offse...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3108 KAFKA-5247: Materialize committed offsets in offset order With this patch, offset commits are always materialized according to the order of the commit records in the offsets topic. Before this patch, transactional offset commits were materialized in transaction order. However, the log cleaner will always preserve the record with the greatest offset. This meant that if there was a mix of offset commits from a consumer and a transactional producer, then it we would switch from transactional order to offset order after cleaning, resulting in an inconsistent state. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5247-materialize-committed-offsets-in-offset-order Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3108 commit f6efd565023d440c6d15091609442ff61ad6f85a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-19T18:31:48Z KAFKA-5247 materialize offset commits in offset order Updated the GroupMetadata to keep track of the offset in the __consumer_offsets topic for the commit record for a given offset commit. We only update the offsets cache when a given offset is committed if the offset of the commit record in the offsets topic is greater than the offset of the existing materialized offset. This way, if we have a mix of transactional and non transactional offset commits for the same group, we will always materialize the offset commtis in offset order. commit 20ee45422130f197791600891a9872826d510ca7 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-19T22:35:27Z Update the return values of the GroupMetadata.remove* methods commit 2fd79d1680711cdd746233dfbeaea957e65e67d8 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-19T23:49:08Z Minor cleanups and added unit tests commit 7e5f2820809d9a085333e1fa97efd13207e5a4e0 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-20T00:02:13Z Remove erroneous comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3094: KAFKA-5269: Correct handling of UNKNOWN_TOPIC_OR_P...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3094 KAFKA-5269: Correct handling of UNKNOWN_TOPIC_OR_PARTITION error We should retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when receiving an UNKNOWN_TOPIC_OR_PARTITION error. As described in the JIRA: It turns out that the `UNKNOWN_TOPIC_OR_PARTITION` is returned from the request handler in KafkaAPis for the AddPartitionsToTxn and the TxnOffsetCommitRequest when the broker's metadata doesn't contain one or more partitions in the request. This can happen for instance when the broker is bounced and has not received the cluster metadata yet. We should retry in these cases, as this is the model followed by the consumer when committing offsets, and by the producer with a ProduceRequest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5269-handle-unknown-topic-partition-in-transaction-manager Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3094.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3094 commit da2e3af528540f73d6d0a35c4c51b8a8dc7eef0d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-18T23:01:33Z Retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when receiving an UNKNOWN_TOPIC_OR_PARTITION error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3093: HOTFIX: Close transactional producers in all new t...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3093 HOTFIX: Close transactional producers in all new tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka HOTFIX-close-leaked-producers-in-transactions-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3093.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3093 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3091: KAFKA-5033: Set default retries for the idempotent...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3091 KAFKA-5033: Set default retries for the idempotent producer to be infinite. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5033-bump-retries-for-idempotent-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3091.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3091 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #3015: KAFKA-5213; Mark a MemoryRecordsBuilder as full as...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3015 KAFKA-5213; Mark a MemoryRecordsBuilder as full as soon as the append stream is closed You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5213-illegalstateexception-in-ensureOpenForAppend Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3015.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3015 commit 799fd8d3d60e3f8950bbe4b7d5e8865e6755f5aa Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-10T22:35:52Z Mark a MemoryRecordsBuilder as full as soon as the append stream is closed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2994: KAFKA-5188: Integration tests for transactions
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2994 KAFKA-5188: Integration tests for transactions You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5188-exactly-once-integration-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2994.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2994 commit b687f143505554b32cd5de05a309a55547496162 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-04-27T06:17:07Z Add transactions integration tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2970: Kafka-5160; KIP-98 Broker side support for TxnOffs...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2970 Kafka-5160; KIP-98 Broker side support for TxnOffsetCommitRequest This patch adds support for the `TxnOffsetCommitRequest` added in KIP-98. Desired handling for this request is [described here](https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.55yzhvkppi6m) . The functionality includes handling the stable state of receiving `TxnOffsetCommitRequests` and materializing results only when the commit marker for the transaction is received. It also handles partition emigration and immigration and rebuilds the required data structures on these events. Tests are included for the basic stable state functionality. Still need to add tests for the immigration/emigration functionality. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-5160-broker-side-support-for-txnoffsetcommitrequest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2970.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2970 commit 925e104ca60bccb6a57ba96c1aaa4135f8734dab Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-02T22:07:04Z WIP commit 27760828c7672b31050cc0c02e9e6e3f6256a0db Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-03T06:59:32Z WIP commit: 1. Able to write txn offset commits to the log and delay materialization until the commit marker appears. 2. Able to recover these commits on partition load. Todo: 1. Tests for the above. 2. Materialize or drop cached offset commits when the transaction marker is received. commit 00ab335cc0423c3544b052cccb5cb47e0e42dcc8 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-03T07:06:30Z Small simplification commit f8139073e7c25781b521c855420ba83b5a1c252a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-03T07:07:54Z fix indentation commit 4a4bbd39ed71edc04ecd31c5000a1b83c73483d3 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-03T23:03:05Z Code complete barring integration. Now to add unit tests commit 8993b93d2e9565016d9be528b7451e1c0a865a35 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-04T01:00:25Z Added the first test cases commit 684ccea1b35a44c19e148f9c9b44cebd5bcb4fa9 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-04T01:16:10Z Completed the functional test cases --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2961: MINOR: Serialize the real isolationLevel in FetchR...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2961 MINOR: Serialize the real isolationLevel in FetchRequest You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka MINOR-serialize-isolation-level-in-fetch-request Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2961.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2961 commit 1caddec12bb12af61dec2fe6909bf04527a9e351 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-02T22:12:35Z Serialize the real isolationLevel in FetchRequest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2840: KAFKA-4818: Exactly once transactional clients
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2840 KAFKA-4818: Exactly once transactional clients You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka exactly-once-transactional-clients Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2840.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2840 commit 8679628642cd0227a1ff1c81a316a11766230e1b Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-30T22:13:11Z CPKAFKA-465 : Implement READ_COMMITTED mode in the KafkaConsumer (#145) commit 477f45c560e681cf65ecc72e3e61eba073ba971b Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-11T18:12:26Z Complete implementation of the transactional producer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2825: KAFKA-5043 : Add FindCoordinatorRPC stub and updat...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2825 KAFKA-5043 : Add FindCoordinatorRPC stub and update InitPidRequest for transactions in KIP-98. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka exactly-once-rpc-stubs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2825.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2825 commit 75ab770af1aabff388fe92db75b26207fda9f029 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-07T22:13:03Z KAFKA-5043 : Add FindCoordinatorRPC stub and update InitPidRequest for transactions in KIP-98. Update initpidrequest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2809: CPKAFKA-465 : Exactly once transactional producer ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2809 CPKAFKA-465 : Exactly once transactional producer -- initial implementation The basic flow works: find coordinator, get pid, add partitions to transaction, commit/abort. Still to add: 1. 'sendOffsets' implementation. 2. error handling. 3. failure test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka exactly-once-transactional-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2809.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2809 commit 40fa5f01866da411f1631c520ac6400a6ea9b6ba Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-03-02T01:42:49Z Transaction log message format (#134) * add transaction log message format * add transaction timeout to initPid request * collapse to one message type commit 1b2df91b19dacf2cd8a5548c86b227104e8bbc67 Author: hachikuji <ja...@confluent.io> Date: 2017-03-07T00:40:53Z Implement FindCoordinatorRequest for transaction coordinator (#140) commit 7a681649ec14e4b00df734fe584bd9dbb955e379 Author: hachikuji <ja...@confluent.io> Date: 2017-03-08T07:00:19Z Exactly once transactions request types (#141) commit a9c32758592112750569f7fd41b538e7500dc959 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-15T20:47:25Z Fix build and test errors due to reabse onto idempotent-producer branch commit 20701f17c20e8121d125bdc11008fb9407f54d11 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-03-17T05:40:49Z Transaction log partition Immigration and Emigration (#142) * sub-package transaction and group classes within coordinator * add loading and cleaning up logic * add transaction configs commit 711657e3b3dead9c7612965def930d01daea6592 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-03-21T04:38:35Z Add transactions broker configs (#146) * add all broker-side configs * check for transaction timeout value * added one more exception type commit bb5c14fd2473abc1b340fcd6d3c5d0dbd787f4a4 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-30T22:13:11Z CPKAFKA-465 : Implement READ_COMMITTED mode in the KafkaConsumer (#145) commit 109ee0e4e7605f92885f6f1994c4efddb42bcc50 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-03-31T22:20:05Z Handle addPartitions and addOffsets on TC (#147) * handling add offsets to txn * add a pending state with prepareTransition / completeTransaction / abortTransition of state * refactor handling logic for multiple in-flight requests commit 846ea79a109e3cb9912832c2a9ed77700b066ef7 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-03T21:17:25Z Fix test errors after rebase: 1. Notable conflicts are with the small API changes to DelayedOperation and the newly introduced purgeDataBefore PR. 2. Jason's update to support streaming decompression required a bit of an overhaul to the way we handle aborted transactions on the consumer. commit b02ae405118cae0385c12ba0abb7e935043752c1 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-17T19:04:15Z Initial plumbing for transactional producer. commit e6b8be80415b525a2b006171bebc8c10ecbe3d9e Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-05T04:46:00Z Initial implementation of the transactional producer commit 413696955bda029019f914e6f8bc758912a59f2d Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-05T06:35:07Z Initial commit of transactional producer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2809: CPKAFKA-465 : Exactly once transactional producer ...
Github user apurvam closed the pull request at: https://github.com/apache/kafka/pull/2809 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2796: Close the producer batch data stream when the batc...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2796 Close the producer batch data stream when the batch gets full to free up compression buffers, etc. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka idempotent-producer-close-data-stream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2796.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2796 commit 8e0b3eba9bde8fb790069ebfe43c3f82f91af4a3 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-04-02T00:53:20Z Close the underlying ProducerBatch data stream once it is closed. This frees up compression buffers, etc. The actual batch will be closed and headers written when the batch is it is about to be sent, or if it is expired, or if the producer is closed. commit e5b0771b74bb576ee1821a7a71bbcc5377b8dcde Author: Ismael Juma <ism...@juma.me.uk> Date: 2017-04-03T14:23:06Z A few improvements --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2735: KAFKA-4815 : Add idempotent producer semantics
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2735 KAFKA-4815 : Add idempotent producer semantics This is from the KIP-98 proposal. The main points of discussion surround the correctness logic, particularly the Log class where incoming entries are validated and duplicates are dropped, and also the producer error handling to ensure that the semantics are sound from the users point of view. There is some subtlety in the idempotent producer semantics. This patch only guarantees idempotent production upto the point where an error has to be returned to the user. Once we hit a such a non-recoverable error, we can no longer guarantee message ordering nor idempotence without additional logic at the application level. In particular, if an application wants guaranteed message order without duplicates, then it needs to do the following on the error callback: # Close the producer so that no queued batches are sent. This is important for guaranteeing ordering. # Read the tail of the log to inspect the last message committed. This is important for avoiding duplicates. You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka exactly-once-idempotent-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2735 commit 83f5b2937d368efa768be6d7df1f57d99425a252 Author: fpj <f...@apache.org> Date: 2016-10-10T16:19:52Z Add broker producer id mapping support commit 37eac83364a67eebf0e1d96552d9fe864baa3d76 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-02-14T19:12:58Z KEOS: idempotent producer pid generation (#126) Add the transaction coordinator for pid generation and management. commit 6d918d0b44302aa81627f3f2748756356614f228 Author: hachikuji <ja...@confluent.io> Date: 2017-02-23T18:59:46Z Minor fixes for test failures and consistency (#133) commit d1868602be8b3e81d98644d673a73b9f730ad586 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-03-01T22:40:30Z Fix headers, new checkstyle rules, and some breakage in KafkaApis commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2 Author: hachikuji <ja...@confluent.io> Date: 2017-03-07T00:23:11Z Avoid removal of non-expired PIDs when log cleaning (#138) commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-09T01:50:01Z Client side implementation of the idempotent producer. (#129) commit 439f284fd4efa56449d43f2fb5e14614689c7b80 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-09T21:23:55Z Fix build errors due to rebase commit 0a81a40258da29ed62caf745e7c1895ca7eead4c Author: hachikuji <ja...@confluent.io> Date: 2017-03-10T17:50:01Z PIDs should be expired according to the transactional id expiration setting (#139) commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-15T18:45:07Z Fix build errors due to rebase commit 848137a28b02cd085605e92726d0457b0153e1c4 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-16T18:43:29Z Remove depependence of MemoryRecordsBuilder on TransactionState commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac Author: hachikuji <ja...@confluent.io> Date: 2017-03-23T22:30:39Z A few minor improvements (#150) commit fafffa8f2445e2a559f24e5286c948c616287f9a Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-24T20:02:48Z Reset transaction state on all irrecoverable exceptions (#153) commit 637f864887e341c51f4fd7192016d2afc45bcf1e Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-24T21:08:29Z Fix build errors due to rebase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2677: MINOR: set trace logging for zookeeper upgrade tes...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2677 MINOR: set trace logging for zookeeper upgrade test This adds logging which will hopefully help root cause https://issues.apache.org/jira/browse/KAFKA-4574. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka minor-set-trace-logging-for-zookeeper-upgrade-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2677.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2677 commit f643d8ddc2b9c9952fa2cd62ba5e3dadac092562 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-11T23:43:48Z MINOR: configure TRACE logging for zookeeper upgrade test We need trace logging for the controller and state change log to debug KAFKA-4574. commit f9c8c51cfdf7eae1842c441ece56d65738cfdfc8 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-12T00:22:52Z More experiments commit 88314b97a774d1c629409ca0e46f596e8f4e7a11 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-12T00:29:25Z Add the trace directory for collection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2567: MINOR: Increase consumer init timeout in throttlin...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2567 MINOR: Increase consumer init timeout in throttling test The throttling system test sometimes fail because it takes longer than the current 10 second time out for partitions to get assigned to the consumer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka increase-timeout-for-partitions-assigned Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2567.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2567 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2347: Initial commit of partition assignment check in co...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2347 Initial commit of partition assignment check in console consumer. With this patch, the consumer will considered initialized in the ProduceConsumeValidate tests only if it has partitions assigned. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-4588-fix-race-between-producer-consumer-start Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2347.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2347 commit ea68caf42c7e8811163583a44b0d7fb1d0150514 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-01-10T23:02:11Z Initial commit of partition assignment check in console consumer as a proxy for the consumer being initalized. With this patch, the consumer will considered initialized in the ProduceConsumeValidate tests only if it has partitions assigned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2278: KAFKA-4526 - Disable throttling test until it can ...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2278 KAFKA-4526 - Disable throttling test until it can be fixed correctly. At present, the test is fragile in the sense that the console consumer has to start and be initialized before the verifiable producer begins producing in the produce-consume-validate loop. If this doesn't happen, the consumer will miss messages at the head of the log and the test will fail. At present, the consumer is considered inited once it has a PID. This is a weak assumption. The plan is to poll appropriate metrics (like partition assignment), and use those as a proxy for consumer initialization. That work will be tracked in a separate ticket. For now, we will disable the tests so that we can get the builds healthy again. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-4526-throttling-test-failures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2278.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2278 commit d7a0e0b9b69e52ca222f18409b3edb6663db0135 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-12-19T22:19:29Z KAFKA-4526 - Disable throttling test until it can be fixed correctly. At present, the test is fragile in the sense that the console consumer has to start and be initialized before the verifiable producer begins producing in the produce-consume-validate loop. If this doesn't happen, the consumer will miss messages at the head of the log and the test will fail. At present, the consumer is considered inited once it has a PID. This is a weak assumption. The plan is to poll appropriate metrics (like partition assignment), and use those as a proxy for consumer initialization. That work will be tracked in a separate ticket. For now, we will disable the tests so that we can get the builds healthy again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #2089: Fix documentation of compaction
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2089 Fix documentation of compaction Also cleaned up some of the language around compaction guarantees. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka fix-documentation-of-compaction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2089.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2089 commit 0af1a864dda32cbf58d270c935dc5e75a38b7d18 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-11-01T22:21:30Z MINOR: fix duplicate line in docs for compaction. commit 03c5bddced47719178117e8c9e2b5b23f472e085 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-11-01T22:27:35Z Fix line length to be consistent with the rest of the file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1910: KAFKA-4214:kafka-reassign-partitions fails all the...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/1910 KAFKA-4214:kafka-reassign-partitions fails all the time when brokers are bounced during reassignment There is a corner case bug, where during partition reassignment, if the controller and a broker receiving a new replica are bounced at the same time, the partition reassignment is failed. The cause of this bug is a block of code in the KafkaController which fails the reassignment if the aliveNewReplicas != newReplicas, ie. if some of the new replicas are offline at the time a controller fails over. The fix is to have the controller listen for ISR change events even for new replicas which are not alive when the controller boots up. Once the said replicas come online, they will be in the ISR set, and the new controller will detect this, and then mark the reassignment as successful. Interestingly, the block of code in question was introduced in KAFKA-990, where a concern about this exact scenario was raised :) This bug was revealed in the system tests in https://github.com/apache/kafka/pull/1904. The relevant tests will be enabled in either this or a followup PR when PR-1904 is merged. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka KAFKA-4214 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1910.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1910 commit 7f9814466592f549e8fcde914815c35dcb5a046d Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-09-26T21:11:44Z There is a corner case bug, where during partition reassignment, if the controller and a broker receiving a new replica are bounced at the same time, the partition reassignment is failed. The cause of this bug is a block of code in the KafkaController which fails the reassignment if the aliveNewReplicas != newReplicas, ie. if some of the new replicas are offline at the time a controller fails over. The fix is to have the controller listen for ISR change events even for new replicas which are not alive when the controller boots up. Once the said replicas come online, they will be in the ISR set, and the new controller will detect this, and then mark the reassignment as successful. Interestingly, the block of code in question was introduced in KAFKA-990, where a concern about this exact scenario was raised :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1904: KAFKA-4213: First set of system tests for replicat...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/1904 KAFKA-4213: First set of system tests for replication throttling, KIP-73. This patch also fixes the following: 1. KafkaService.verify_reassign_partitions did not check whether partition reassignment actually completed successfully (KAFKA-4204). This patch works around those shortcomings so that we get the right signal from this method. 2. ProduceConsumeValidateTest.annotate_missing_messages would call `pop' on the list of missing messages, causing downstream methods to get incomplete data. We fix that in this patch as well. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka throttling-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1904.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1904 commit fe4a0b1070f25e687fb8075210da9c5a356fa1c8 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-09-23T23:41:02Z Initial commit of system tests for replication throttling, KIP-73. This patch also fixes the following: 1. KafkaService.verify_reassign_partitions did not check whether partition reassignment actually completed successfully (KAFKA-4204). This patch works around those shortcomings so that we get the right signal from this method. 2. ProduceConsumeValidateTest.annotate_missing_messages would call `pop' on the list of missing messages, causing downstream methods to get incomplete data. We fix that in this patch as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1903: KAFKA-4213: First set of system tests for replicat...
Github user apurvam closed the pull request at: https://github.com/apache/kafka/pull/1903 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1903: KAFKA-4213: First set of system tests for replicat...
GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/1903 KAFKA-4213: First set of system tests for replication throttling Added the first set of system tests for replication quotas. These tests validate throttling behavior during partition reassigment. Along with this patch are fixes to the test framework which include: 1. KakfaService.verify_replica_reassignment: this method was a no-op and would always return success, as explained in KAFKA-4204. This patch adds a workaround to the problems mentioned there, by grepping correctly for success, failure, and 'in progress' states of partition reassignment. 2.ProduceConsumeValidateTest.annotate_missing_messages would call missing.pop() to enumerate the first 20 missing messages. This meant that all future counts of what is actually missing would be off by 20, leading to the impression of data loss. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apurvam/kafka throttling-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1903.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1903 commit 372db72c2da55dd2aba70019b258429855832804 Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-09-09T18:53:42Z Merge remote-tracking branch 'apache/trunk' into trunk commit ae912d444d3fb63c2e5487f88949408e0b1207e9 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-09T20:44:55Z KAFKA-3807; Fix transient test failure caused by race on future completion Author: Jason Gustafson <ja...@confluent.io> Reviewers: Dan Norwood <norw...@confluent.io>, Ismael Juma <ism...@juma.me.uk> Closes #1821 from hachikuji/KAFKA-3807 commit d0a86ffdec330f6e7213a370287a2d81bb93e2bc Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Date: 2016-09-10T07:16:23Z KAFKA-4145; Avoid redundant integration testing in ProducerSendTests Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1842 from vahidhashemian/KAFKA-4145 commit 42b5583561895e308063ed9e2186d83c83ca35d8 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-11T07:46:20Z KAFKA-4147; Fix transient failure in ConsumerCoordinatorTest.testAutoCommitDynamicAssignment Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1841 from hachikuji/KAFKA-4147 commit e7697ad0ab0f292ad1e29d9a159d113574bfcf67 Author: Eric Wasserman <eric.wasser...@gmail.com> Date: 2016-09-12T01:45:05Z KAFKA-1981; Make log compaction point configurable Now uses LogSegment.largestTimestamp to determine age of segment's messages. Author: Eric Wasserman <eric.wasser...@gmail.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #1794 from ewasserman/feat-1981 commit b36034eaa4eb284fafddb1a7507a2cf187993e62 Author: Damian Guy <damian@gmail.com> Date: 2016-09-12T04:00:32Z MINOR: catch InvalidStateStoreException in QueryableStateIntegrationTest A couple of the tests may transiently fail in QueryableStateIntegrationTest as they are not catching InvalidStateStoreException. This exception is expected during rebalance. Author: Damian Guy <damian@gmail.com> Reviewers: Eno Thereska, Guozhang Wang Closes #1840 from dguy/minor-fix commit 642b709f919a02379f9d0c9313586b02d179ca78 Author: Tim Brooks <t...@uncontended.net> Date: 2016-09-13T03:28:01Z KAFKA-2311; Make KafkaConsumer's ensureNotClosed method thread-safe Here is the patch on github ijuma. Acquiring the consumer lock (the single thread access controls) requires that the consumer be open. I changed the closed variable to be volatile so that another thread's writes will visible to the reading thread. Additionally, there was an additional check if the consumer was closed after the lock was acquired. This check is no longer necessary. This is my original work and I license it to the project under the project's open source license. Author: Tim Brooks <t...@uncontended.net> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1637 from tbrooks8/KAFKA-2311 commit ca539df5887bdfdbe86ba45f5514ed54b3b648d4 Author: Dong Lin <lindon...@gmail.com> Date: 2016-09-14T00:33:54Z KAFKA-4158; Reset quota to default value if quota override is deleted Author: Dong Lin <lindon...@gmail.com> Reviewers: Joel Koshy <jjkosh...@gmail.com>, Jiangjie Qin <becket@gmail.com> Closes #1851 from lindong28/K
