[
https://issues.apache.org/jira/browse/KAFKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832805#comment-15832805
]
ASF GitHub Bot commented on KAFKA-4547:
---------------------------------------
GitHub user vahidhashemian opened a pull request:
https://github.com/apache/kafka/pull/2415
KAFKA-4547 (0.10.1 hotfix): Avoid unnecessary offset commit that could lead
to an invalid offset position if partition is paused
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vahidhashemian/kafka KAFKA-4547-0.10.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2415.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2415
----
commit 6acbf27aebbe5b6b4f230ce44ea5151c4c53ffbd
Author: Jason Gustafson <[email protected]>
Date: 2016-09-20T02:59:03Z
MINOR: Bump to version 0.10.2
commit f396fdac197409fb955f00a6f642f04e4926ba41
Author: Ben Stopford <[email protected]>
Date: 2016-09-20T06:17:23Z
KAFKA-4193; Fix for intermittent failure in FetcherTest
Author: Ben Stopford <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #1881 from benstopford/KAFKA-4193
commit c195003cb6e05f2d8c49285ff7e77b1cb3aa4361
Author: Eno Thereska <[email protected]>
Date: 2016-09-20T10:33:50Z
HOTFIX: Added check for metadata unavailable
Author: Eno Thereska <[email protected]>
Reviewers: Damian Guy <[email protected]>, Ismael Juma
<[email protected]>
Closes #1887 from enothereska/hotfix-metadata-unavailable
commit 3663275cf066b7715cc11b26fd9c144bbff1c373
Author: Ben Stopford <[email protected]>
Date: 2016-09-20T13:53:48Z
KAFKA-4184; Intermittent failures in
ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle
Build is unstable, so it's hard to validate this change. Of the various
builds up until 11am BST the test ran twice and passed twice.
Author: Ben Stopford <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1873 from benstopford/KAFKA-4184
commit 4f821830bc6b726cddf90999fff76006745b1a3f
Author: Ben Stopford <[email protected]>
Date: 2016-09-20T14:41:14Z
KAFKA-4197; Make ReassignPartitionsTest System Test move data
The ReassignPartitionsTest system tests doesn't reassign any replicas (i.e.
move data).
This is a simple issue. It uses a 3 node cluster with replication factor of
3, so whilst the replicas are jumbled around, nothing actually is moved from
machine to machine when the assignment is executed.
This fix just ups the number of nodes to 4 so things move.
Tests pass locally.
There are runs pending on the two branch builders
Passes:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/551/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/94/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/553/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/554/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/95
Failures:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/552 =>
_RuntimeError: There aren't enough available nodes to satisfy the resource
request. Total cluster size: 1, Requested: 4, Already allocated: 1, Available:
0._ Which I assume to do with the test env.
Author: Ben Stopford <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1892 from benstopford/fix_reassignment_test
commit 24f81ea764a493b4422b6a3ef6b3e771d0e4d63b
Author: Damian Guy <[email protected]>
Date: 2016-09-21T18:11:12Z
MINOR: add javadoc comment to PersistenKeyValueFactory.enableCaching
missing javadoc on public API method PersistenKeyValueFactory.enableCaching
Author: Damian Guy <[email protected]>
Reviewers: Eno Thereska, Guozhang Wang
Closes #1891 from dguy/minor-java-doc
commit a632716a3c9a871f325c6f13aefa9aed0add4b82
Author: Damian Guy <[email protected]>
Date: 2016-09-21T18:13:39Z
MINOR: remove unused code from InternalTopicManager
Remove isValidCleanupPolicy and related fields as they are never used.
Author: Damian Guy <[email protected]>
Reviewers: Eno Thereska, Guozhang Wang
Closes #1888 from dguy/minor-remove-unused
commit 732fabf94ebc9631d31f2feb2116ee8b63beabef
Author: Jason Gustafson <[email protected]>
Date: 2016-09-22T17:07:50Z
KAFKA-3782: Ensure heartbeat thread restarted after rebalance woken up
Author: Jason Gustafson <[email protected]>
Reviewers: Guozhang Wang
Closes #1898 from hachikuji/KAFKA-3782
commit 27e3edc791760dea7ff4d048f87d1585f9e235d7
Author: Elias Levy <[email protected]>
Date: 2016-09-22T17:33:23Z
MINOR: Fix comments in KStreamKStreamJoinTest
Minor comment fixes.
Author: Elias Levy <[email protected]>
Reviewers: Guozhang Wang
Closes #1885 from eliaslevy/fix-test-comments
commit 4112bc3af7bec63ae1248c1b9db745ba8ac4c63c
Author: Eno Thereska <[email protected]>
Date: 2016-09-22T19:45:43Z
HOTFIX: Decrease commit interval
The original commit interval of 30 seconds might be too large in some
cases, e.g., when the verifier finishes before those 30 seconds have elapsed.
Author: Eno Thereska <[email protected]>
Reviewers: Damian Guy, Guozhang Wang
Closes #1899 from enothereska/hotfix-smoke-test-commit-interval
commit d32f3f2828c199a5a09bbba779d07430e43fb190
Author: Ismael Juma <[email protected]>
Date: 2016-09-22T21:53:40Z
MINOR: Increase `zkConnectionTimeout` and timeout in `testReachableServer`
We had a number of failures recently due to these timeouts being too low.
It's a particular problem if multiple forks are used while running the tests.
Author: Ismael Juma <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #1889 from ijuma/increase-zk-timeout-in-tests
commit 106a7456060750ab0604d290b8c1e33a57adbf20
Author: Jason Gustafson <[email protected]>
Date: 2016-09-22T22:03:49Z
MINOR: Add test cases for delays in consumer rebalance listener
Author: Jason Gustafson <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1866 from hachikuji/rebalance-delay-test-cases
commit 6a13a3dbaddf99850b2583007577fa2a6e1e6d3a
Author: Jason Gustafson <[email protected]>
Date: 2016-09-23T20:13:29Z
KAFKA-3590; Handle not-enough-replicas errors when writing to offsets topic
Author: Jason Gustafson <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Guozhang Wang
<[email protected]>
Closes #1859 from hachikuji/KAFKA-3590
commit 0de807357be4d776747e960b5fd17ddeae9a6cad
Author: Ryan Pridgeon <[email protected]>
Date: 2016-09-23T21:20:41Z
KAFKA-3719; Allow underscores in hostname
Technically this does not strictly adhere to RFC-952 however it is valid
for domain names, urls and uris so we should loosen the requirements a tad.
Author: Ryan Pridgeon <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1856 from rnpridgeon/KAFKA-3719
commit 36242b846a42b33d7d4c1931f2dae93ebe1547c7
Author: Sumit Arrawatia <[email protected]>
Date: 2016-09-24T09:16:49Z
KAFKA-4151; Update public docs for Cluster Id (KIP-78)
- Updated implementation docs with details on Cluster Id generation.
- Mention cluster id in "noteworthy changes for 0.10.1.0" in upgrade docs.
Author: Sumit Arrawatia <[email protected]>
Author: arrawatia <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1895 from arrawatia/kip-78-docs
commit 1d055f7551d138324d2540095a1cfc1c8f74d76f
Author: Arun Mahadevan <[email protected]>
Date: 2016-09-25T07:44:56Z
KAFKA-3282; Change tools to use new consumer if zookeeper is not specified
Author: Arun Mahadevan <[email protected]>
Reviewers: Jason Gustafson <[email protected]>, Ismael Juma
<[email protected]>
Closes #1376 from arunmahadevan/cons-consumer-fix
commit c0a62b70a8eadc550c937bb18e0203ab691618f5
Author: Rajini Sivaram <[email protected]>
Date: 2016-09-26T00:01:45Z
KAFKA-4055; System tests for secure quotas
Fix existing client-id quota test which currently don't configure quota
overrides correctly. Add new tests for user and (user, client-id) quota
overrides and default quotas.
Author: Rajini Sivaram <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1860 from rajinisivaram/KAFKA-4055
commit b75245cfbbefc712103b9329da0f27a205baa6aa
Author: Jason Gustafson <[email protected]>
Date: 2016-09-26T21:54:01Z
MINOR: Wakeups propagated from commitOffsets in WorkerSinkTask should be
caught
Author: Jason Gustafson <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>
Closes #1907 from hachikuji/catch-wakeup-worker-sink-task
commit be6056abc9970600347c95c4c8658799b76dbe6b
Author: Apurva Mehta <[email protected]>
Date: 2016-09-27T00:18:18Z
KAFKA-4214; kafka-reassign-partitions fails all the time when brokers are
bounced during reassignment
There is a corner case bug, where during partition reassignment, if the
controller and a broker receiving a new replica are bounced at the same
time, the partition reassignment is failed.
The cause of this bug is a block of code in the KafkaController which
fails the reassignment if the aliveNewReplicas != newReplicas, ie. if
some of the new replicas are offline at the time a controller fails
over.
The fix is to have the controller listen for ISR change events even for
new replicas which are not alive when the controller boots up. Once the
said replicas come online, they will be in the ISR set, and the new
controller will detect this, and then mark the reassignment as
successful.
Interestingly, the block of code in question was introduced in
KAFKA-990, where a concern about this exact scenario was raised :)
This bug was revealed in the system tests in
https://github.com/apache/kafka/pull/1904.
The relevant tests will be enabled in either this or a followup PR when
PR-1904 is merged.
Thanks to junrao identifying the issue and providing the patch.
Author: Apurva Mehta <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1910 from apurvam/KAFKA-4214
commit 60ad6d727861a87fa756918a7be7547e9b1f4c3d
Author: Vahid Hashemian <[email protected]>
Date: 2016-09-27T02:04:22Z
KAFKA-3831; Prepare for updating new-consumer-based Mirror Maker's default
partition assignment strategy to round robin
This patch adds proper warning message and necessary doc updates for
updating the default partition assignment strategy of Mirror Maker from range
to round robin. The actual switch would occur as part of a major release cycle
(to be scheduled).
Author: Vahid Hashemian <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #1499 from vahidhashemian/KAFKA-3831
commit a208478f37f216324a87b530f18f30ecd79601c8
Author: Magnus Reftel <[email protected]>
Date: 2016-09-27T04:15:25Z
MINOR: Fix Javadoc of @throws AuthorizationException for KafkaConsumer.poll
Author: Magnus Reftel <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #1901 from reftel/feature/poll_javadoc
commit 5d6408f6cfda3f8ab366195f69e90de048cde25d
Author: Ben Stopford <[email protected]>
Date: 2016-09-27T13:00:44Z
KAFKA-4200; Fix throttle argument in kafka-reassign-partitions.sh
Simple jira which alters two things:
1. kafka-reassign-partitions --verify prints Throttle was removed
regardless of whether a throttle was applied. It should only print this if the
value was actually changed.
2. --verify should exception if the —throttle argument. (check generate too)
To test this I extracted all validation logic into a separate method and
added a test which covers the majority of combinations. The validation logic
was retained as is, other than implementing (2) and adding validation to the
--broker-list option which you can currently apply to any of hte main actions
(where it is ignored). Requirement 1 was tested manually (as it's just println).
Testing:
- Build passes locally.
- System test reassign_partitions_test.py also passes.
Author: Ben Stopford <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #1896 from benstopford/KAFKA-4200
commit b8ed4a51134ca8b98d1445871a55ed33b6ad5b92
Author: Ben Stopford <[email protected]>
Date: 2016-09-27T19:27:20Z
KAFKA-4177; Remove ThrottledReplicationRateLimit from Server Config
This small PR pulls ThrottledReplicationRateLimit out of KafkaConfig and
puts it in a class that defines Dynamic Configs. Client configs are also placed
in this class and validation added.
Author: Ben Stopford <[email protected]>
Reviewers: Jun Rao <[email protected]>, Ismael Juma <[email protected]>
Closes #1864 from benstopford/KAFKA-4177
commit 3db752a565071c78e4b11eaafa739844fa785b04
Author: Jason Gustafson <[email protected]>
Date: 2016-09-27T20:35:29Z
MINOR: Make new consumer default for Mirror Maker
Author: Jason Gustafson <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1914 from hachikuji/mm-default-new-consumer
commit 61d3378bc84914a521a65cdfffb7299928fa8671
Author: Ismael Juma <[email protected]>
Date: 2016-09-27T23:49:46Z
MINOR: Remove no longer required --new-consumer switch in docs
Author: Ismael Juma <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #1905 from ijuma/no-new-consumer-switch-in-examples
commit 0c25c73782e6e70b8f37e3dda2fa2a5b0b1c8c65
Author: Damian Guy <[email protected]>
Date: 2016-09-28T00:35:24Z
HOTFIX: fix npe in StreamsMetadataState when onChange has not been called
If some StreamsMetadataState methods are called before the onChange method
is called a NullPointerException was being thrown. Added null check for cluster
in isInitialized method
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1920 from dguy/fix-npe-streamsmetadata
commit 0c4cc5a4466d0dfce01903c29d97af3e428597f3
Author: Guozhang Wang <[email protected]>
Date: 2016-09-28T00:37:12Z
MINOR: fixes a few error logging formats
Author: Guozhang Wang <[email protected]>
Reviewers: Eno Thereska <[email protected]>
Closes #1919 from guozhangwang/minor-error-message-fixes
commit c526c0c3f6f153ef8f84a3e66d9f997d8c31e20c
Author: Damian Guy <[email protected]>
Date: 2016-09-28T00:43:36Z
MINOR: add test to make sure ProcessorStateManager can handle State Stores
with logging disabled
Adding the test so we know that the State Stores with logging disabled or
without a topic don't throw any exceptions.
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1916 from dguy/state-store-logging-disabled
commit d83cde7cabe4e86951c6760e68e65b99752cfe0e
Author: Damian Guy <[email protected]>
Date: 2016-09-28T18:07:44Z
KAFKA-3708: Better exception handling in Kafka Streams
KafkaExceptions currently thrown from within StreamThread/StreamTask
currently bubble up without any additional context. This makes it hard to
figure out where something went wrong, i.e, which topic had the serialization
exception etc
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #1819 from dguy/kafka-3708 and squashes the following commits:
d6feaa8 [Damian Guy] address comments
15b89e7 [Damian Guy] merge trunk
6b8a8af [Damian Guy] catch exceptions in various places and throw more
informative versions
c86eeda [Damian Guy] fix conflicts
8f37e2c [Damian Guy] add some context to exceptions
commit 281fac9ed626b16190d56e609c9bf04a3188374a
Author: Rajini Sivaram <[email protected]>
Date: 2016-09-28T19:18:20Z
KAFKA-4227; Shutdown AdminManager when KafkaServer is shutdown
Terminate topic purgatory thread in AdminManager during server shutdown to
avoid threads being left around in unit tests.
Author: Rajini Sivaram <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #1927 from rajinisivaram/KAFKA-4227
----
> Consumer.position returns incorrect results for Kafka 0.10.1.0 client
> ---------------------------------------------------------------------
>
> Key: KAFKA-4547
> URL: https://issues.apache.org/jira/browse/KAFKA-4547
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.10.1.0, 0.10.0.2, 0.10.1.1
> Environment: Windows Kafka 0.10.1.0
> Reporter: Pranav Nakhe
> Assignee: Vahid Hashemian
> Priority: Blocker
> Labels: clients
> Fix For: 0.10.2.0
>
> Attachments: issuerep.zip
>
>
> Consider the following code -
> KafkaConsumer<String, String> consumer = new
> KafkaConsumer<String, String>(props);
> List<TopicPartition> listOfPartitions = new ArrayList();
> for (int i = 0; i <
> consumer.partitionsFor("IssueTopic").size(); i++) {
> listOfPartitions.add(new TopicPartition("IssueTopic",
> i));
> }
> consumer.assign(listOfPartitions);
> consumer.pause(listOfPartitions);
> consumer.seekToEnd(listOfPartitions);
> // consumer.resume(listOfPartitions); -- commented out
> for(int i = 0; i < listOfPartitions.size(); i++) {
>
> System.out.println(consumer.position(listOfPartitions.get(i)));
> }
>
> I have created a topic IssueTopic with 3 partitions with a single replica on
> my single node kafka installation (0.10.1.0)
> The behavior noticed for Kafka client 0.10.1.0 as against Kafka client
> 0.10.0.1
> A) Initially when there are no messages on IssueTopic running the above
> program returns
> 0.10.1.0
> 0
> 0
> 0
> 0.10.0.1
> 0
> 0
> 0
> B) Next I send 6 messages and see that the messages have been evenly
> distributed across the three partitions. Running the above program now
> returns
> 0.10.1.0
> 0
> 0
> 2
> 0.10.0.1
> 2
> 2
> 2
> Clearly there is a difference in behavior for the 2 clients.
> Now after seekToEnd call if I make a call to resume (uncomment the resume
> call in code above) then the behavior is
> 0.10.1.0
> 2
> 2
> 2
> 0.10.0.1
> 2
> 2
> 2
> This is an issue I came across when using the spark kafka integration for
> 0.10. When I use kafka 0.10.1.0 I started seeing this issue. I had raised a
> pull request to resolve that issue [SPARK-18779] but when looking at the
> kafka client implementation/documentation now it seems the issue is with
> kafka and not with spark. There does not seem to be any documentation which
> specifies/implies that we need to call resume after seekToEnd for position to
> return the correct value. Also there is a clear difference in the behavior in
> the two kafka client implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
