GitHub user ZoneMayor reopened a pull request:
https://github.com/apache/kafka/pull/671
KAFKA-2977: Transient Failure in
kafka.log.LogCleanerIntegrationTest.cleanerTest
Code is as below:
val appends = writeDups(numKeys = 100, numDups = 3, log,
CompressionCodec.getCompressionCodec(compressionCodec))
cleaner.startup()
val firstDirty = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty)
....
val appends2 = appends ++ writeDups(numKeys = 100, numDups = 3, log,
CompressionCodec.getCompressionCodec(compressionCodec))
val firstDirty2 = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty2)
log cleaner and writeDups are two different threads;
log cleaner do cleaning every 15s, timeout in "cleaner.awaitCleaned" is 60s;
there is a filtering condition for a log to be chosen to become a cleaning
target: cleanableRatio> 0.5(configured log.cleaner.min.cleanable.ratio) by
default;
It may happen that, during "val appends2 = appends ++ writeDups(numKeys =
100, numDups = 3, log,
CompressionCodec.getCompressionCodec(compressionCodec))", log is also
undergoing a cleaning process;
Since the segment size configured in this test is quite small: 100, there
is possibility that before the end of 'writeDups', some 'dirty segment' of the
log is already cleaned;
With tiny dirty part left, cleanableRatio> 0.5 cannot be satisfied;
thus firstDirty2>lastCleaned2, which leads to this test failed;
To fix, make MinCleanableDirtyRatioProp configurable in makeCleaner;
Also removed minDirtyMessages.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2977
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/671.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #671
----
commit 95374147a28208d4850f6e73f714bf418935fc2d
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-11-27T03:49:34Z
Merge pull request #1 from apache/trunk
merge
commit cec5b48b651a7efd3900cfa3c1fd0ab1eeeaa3ec
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-01T10:44:02Z
Merge pull request #2 from apache/trunk
2015-12-1
commit a119d547bf1741625ce0627073c7909992a20f15
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-04T13:42:27Z
Merge pull request #3 from apache/trunk
2015-12-04#KAFKA-2893
commit b767a8dff85fc71c75d4cf5178c3f6f03ff81bfc
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-09T10:42:30Z
Merge pull request #5 from apache/trunk
2015-12-9
commit 0070c2d71d06ee8baa1cddb3451cd5af6c6b1d4a
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-11T14:50:30Z
Merge pull request #8 from apache/trunk
2015-12-11
commit 09908ac646d4c84f854dad63b8c99213b74a7063
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-13T14:17:19Z
Merge pull request #9 from apache/trunk
2015-12-13
commit ff1e68bb7101d12624c189174ef1dceb21ed9798
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-13T14:31:34Z
KAFKA-2054: Transient Failure in
kafka.log.LogCleanerIntegrationTest.cleanerTest
commit 6321ab6599cb7a981fac2a4eea64a5f2ea805dd6
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-13T14:36:11Z
removed unnecessary maven repo
commit 05cae52c72a02c0ed40fd4e3be03e1cb19f33f7a
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-17T12:21:12Z
removed the semicolon
commit 34240b52e1b70aa172b65155f6042243d838b420
Author: ZoneMayor <jinxing6...@126.com>
Date: 2015-12-18T07:22:20Z
Merge pull request #12 from apache/trunk
2015-12-18
commit 651de48663cf375ea714cdbeb34650d75f1f4d71
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-18T07:43:38Z
KAFKA-2977: Transient Failure in
kafka.log.LogCleanerIntegrationTest.cleanerTest
commit ba03eee18045dc4aabc56ff17907036c238b1f7d
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-18T07:50:57Z
KAFKA-2977: fix
commit 341c0f7d9d8a6910bc81a1cc9da6608c48323201
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-18T17:08:43Z
Merge branch 'trunk-KAFKA-2977' of https://github.com/ZoneMayor/kafka into
trunk-KAFKA-2977
commit 734d1d5ee179cf8671426ec770c8cab5f6228112
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-18T17:23:15Z
KAFKA-2977: using String interpolation
commit 55430ed040354279f06bd59ff1a62c2a50c8882c
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-19T13:39:39Z
make MinCleanableDirtyRatioProp configurable in makeCleaner and removed
minDirtyMessages
commit 37d3e41f120a16da5571c9082147efb422cfc2d9
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-19T13:40:20Z
Merge branch 'trunk-KAFKA-2977' of https://github.com/ZoneMayor/kafka into
trunk-KAFKA-2977
commit ef7bf83671bd9219e845f45f3bdf5153cdcf0ed9
Author: jinxing <jinx...@fenbi.com>
Date: 2015-12-19T13:42:22Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
