[kafka] branch 2.4 updated: MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6
This is an automated email from the ASF dual-hosted git repository. manikumar pushed a commit to branch 2.4 in repository https://gitbox.apache.org/repos/asf/kafka.git The following commit(s) were added to refs/heads/2.4 by this push: new 5f91aa9 MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6 5f91aa9 is described below commit 5f91aa99da6e8f5dd51a6ea573b547150163495e Author: Manikumar Reddy AuthorDate: Sat Nov 9 14:30:37 2019 +0530 MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6 ZK upgrade from 3.4.X to 3.5.6 fails with "java.io.IOException: No snapshot found" if there are no snapshot files. This was discussed in https://issues.apache.org/jira/browse/ZOOKEEPER-3056 Author: Manikumar Reddy Reviewers: Ismael Juma Closes #7625 from omkreddy/zk-upgrade (cherry picked from commit dbce4c12ba1b5056146ffa9fd5cb1d9f38b986c7) Signed-off-by: Manikumar Reddy --- config/zookeeper.properties | 5 + docs/upgrade.html | 9 + 2 files changed, 14 insertions(+) diff --git a/config/zookeeper.properties b/config/zookeeper.properties index 90f4332..8c186a3 100644 --- a/config/zookeeper.properties +++ b/config/zookeeper.properties @@ -22,3 +22,8 @@ maxClientCnxns=0 # Set the port to something non-conflicting if choosing to enable this admin.enableServer=false # admin.serverPort=8080 + +# Enable snapshot.trust.empty config if the ZK upgrade from 3.4.X to 3.5.6 is failing +# with "java.io.IOException: No snapshot found, but there are log entries" error. +# Check upgrade docs for more details. +# snapshot.trust.empty=true diff --git a/docs/upgrade.html b/docs/upgrade.html index 8bc0fb9..ce894d1 100644 --- a/docs/upgrade.html +++ b/docs/upgrade.html @@ -61,6 +61,15 @@ +Additional Upgrade Notes: + + +ZooKeeper has been upgraded to 3.5.6. Set snapshot.trust.empty=true in zookeeper.properties before the upgrade +to avoid https://issues.apache.org/jira/browse/ZOOKEEPER-3056;>ZOOKEEPER-3056. After the new version starts successfully, it is safe to remove this config. +See https://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#sc_advancedConfiguration;>ZooKeeper advanced configuration for more details. + + + Notable changes in 2.4.0 A new Admin API has been added for partition reassignments. Due to changing the way Kafka propagates reassignment information,
[kafka] branch trunk updated: MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6
This is an automated email from the ASF dual-hosted git repository. manikumar pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/kafka.git The following commit(s) were added to refs/heads/trunk by this push: new dbce4c1 MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6 dbce4c1 is described below commit dbce4c12ba1b5056146ffa9fd5cb1d9f38b986c7 Author: Manikumar Reddy AuthorDate: Sat Nov 9 14:30:37 2019 +0530 MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6 ZK upgrade from 3.4.X to 3.5.6 fails with "java.io.IOException: No snapshot found" if there are no snapshot files. This was discussed in https://issues.apache.org/jira/browse/ZOOKEEPER-3056 Author: Manikumar Reddy Reviewers: Ismael Juma Closes #7625 from omkreddy/zk-upgrade --- config/zookeeper.properties | 5 + docs/upgrade.html | 9 + 2 files changed, 14 insertions(+) diff --git a/config/zookeeper.properties b/config/zookeeper.properties index 90f4332..8c186a3 100644 --- a/config/zookeeper.properties +++ b/config/zookeeper.properties @@ -22,3 +22,8 @@ maxClientCnxns=0 # Set the port to something non-conflicting if choosing to enable this admin.enableServer=false # admin.serverPort=8080 + +# Enable snapshot.trust.empty config if the ZK upgrade from 3.4.X to 3.5.6 is failing +# with "java.io.IOException: No snapshot found, but there are log entries" error. +# Check upgrade docs for more details. +# snapshot.trust.empty=true diff --git a/docs/upgrade.html b/docs/upgrade.html index a3920ce..2d7fa33 100644 --- a/docs/upgrade.html +++ b/docs/upgrade.html @@ -68,6 +68,15 @@ +Additional Upgrade Notes: + + +ZooKeeper has been upgraded to 3.5.6. Set snapshot.trust.empty=true in zookeeper.properties before the upgrade +to avoid https://issues.apache.org/jira/browse/ZOOKEEPER-3056;>ZOOKEEPER-3056. After the new version starts successfully, it is safe to remove this config. +See https://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#sc_advancedConfiguration;>ZooKeeper advanced configuration for more details. + + + Notable changes in 2.4.0 A new Admin API has been added for partition reassignments. Due to changing the way Kafka propagates reassignment information,
[kafka-site] branch asf-site updated: Add committer picture and info for Mickael Maison (#235)
This is an automated email from the ASF dual-hosted git repository. mimaison pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/kafka-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 936eabe Add committer picture and info for Mickael Maison (#235) 936eabe is described below commit 936eabe65c660512180f2988a8e9836bb31f576d Author: Mickael Maison AuthorDate: Sat Nov 9 11:59:09 2019 + Add committer picture and info for Mickael Maison (#235) --- committers.html| 5 + images/mickael.jpg | Bin 0 -> 7892 bytes 2 files changed, 5 insertions(+) diff --git a/committers.html b/committers.html index ab51318..cb38876 100644 --- a/committers.html +++ b/committers.html @@ -269,8 +269,13 @@ https://twitter.com/rhauch;>@rhauch + + Mickael Maison + Committer + https://www.linkedin.com/in/mickaelmaison/;>/in/mickaelmaison + https://twitter.com/MickaelMaison;>@MickaelMaison diff --git a/images/mickael.jpg b/images/mickael.jpg new file mode 100644 index 000..9792378 Binary files /dev/null and b/images/mickael.jpg differ
[kafka] annotated tag 2.4.0-rc1 updated (bd59745 -> 2744f29)
This is an automated email from the ASF dual-hosted git repository. manikumar pushed a change to annotated tag 2.4.0-rc1 in repository https://gitbox.apache.org/repos/asf/kafka.git. *** WARNING: tag 2.4.0-rc1 was modified! *** from bd59745 (commit) to 2744f29 (tag) tagging bd59745880daefb50664d9c58bbe81d0a5f7f7fe (commit) replaces 0.8.0-beta1 by Manikumar Reddy on Sat Nov 9 14:59:50 2019 +0530 - Log - 2.4.0-rc1 --- No new revisions were added by this update. Summary of changes:
[kafka] branch 2.3 updated: KAFKA-9133; Cleaner should handle log start offset larger than active segment base offset (#7662)
This is an automated email from the ASF dual-hosted git repository.
jgus pushed a commit to branch 2.3
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/2.3 by this push:
new 4e5b864 KAFKA-9133; Cleaner should handle log start offset larger
than active segment base offset (#7662)
4e5b864 is described below
commit 4e5b86419982050217d06b3c30ba6236e1fd9090
Author: Jason Gustafson
AuthorDate: Fri Nov 8 19:35:22 2019 -0800
KAFKA-9133; Cleaner should handle log start offset larger than active
segment base offset (#7662)
This was a regression in 2.3.1. In the case of a DeleteRecords call, the
log start offset may be higher than the active segment base offset. The cleaner
should allow for this case gracefully.
Reviewers: Jun Rao
Co-Authored-By: Tim Van Laer
---
core/src/main/scala/kafka/log/Log.scala| 10 +--
core/src/main/scala/kafka/log/LogCleaner.scala | 13 +--
.../main/scala/kafka/log/LogCleanerManager.scala | 43 +++---
.../unit/kafka/log/LogCleanerIntegrationTest.scala | 4 +-
.../unit/kafka/log/LogCleanerManagerTest.scala | 99 --
core/src/test/scala/unit/kafka/log/LogTest.scala | 1 +
6 files changed, 111 insertions(+), 59 deletions(-)
diff --git a/core/src/main/scala/kafka/log/Log.scala
b/core/src/main/scala/kafka/log/Log.scala
index 491a4c4..3291e1d 100644
--- a/core/src/main/scala/kafka/log/Log.scala
+++ b/core/src/main/scala/kafka/log/Log.scala
@@ -1918,8 +1918,8 @@ class Log(@volatile var dir: File,
lock synchronized {
val view = Option(segments.floorKey(from)).map { floor =>
if (to < floor)
- throw new IllegalArgumentException(s"Invalid log segment range:
requested segments from offset $from " +
-s"mapping to segment with base offset $floor, which is greater
than limit offset $to")
+ throw new IllegalArgumentException(s"Invalid log segment range:
requested segments in $topicPartition " +
+s"from offset $from mapping to segment with base offset $floor,
which is greater than limit offset $to")
segments.subMap(floor, to)
}.getOrElse(segments.headMap(to))
view.values.asScala
@@ -1929,9 +1929,9 @@ class Log(@volatile var dir: File,
def nonActiveLogSegmentsFrom(from: Long): Iterable[LogSegment] = {
lock synchronized {
if (from > activeSegment.baseOffset)
-throw new IllegalArgumentException("Illegal request for non-active
segments beginning at " +
- s"offset $from, which is larger than the active segment's base
offset ${activeSegment.baseOffset}")
- logSegments(from, activeSegment.baseOffset)
+Seq.empty
+ else
+logSegments(from, activeSegment.baseOffset)
}
}
diff --git a/core/src/main/scala/kafka/log/LogCleaner.scala
b/core/src/main/scala/kafka/log/LogCleaner.scala
index 7e51ff4..47abae5 100644
--- a/core/src/main/scala/kafka/log/LogCleaner.scala
+++ b/core/src/main/scala/kafka/log/LogCleaner.scala
@@ -435,17 +435,6 @@ object LogCleaner {
fileSuffix = Log.CleanedFileSuffix, initFileSize = log.initFileSize,
preallocate = log.config.preallocate)
}
- /**
-* Given the first dirty offset and an uncleanable offset, calculates the
total cleanable bytes for this log
-* @return the biggest uncleanable offset and the total amount of cleanable
bytes
-*/
- def calculateCleanableBytes(log: Log, firstDirtyOffset: Long,
uncleanableOffset: Long): (Long, Long) = {
-val firstUncleanableSegment =
log.nonActiveLogSegmentsFrom(uncleanableOffset).headOption.getOrElse(log.activeSegment)
-val firstUncleanableOffset = firstUncleanableSegment.baseOffset
-val cleanableBytes = log.logSegments(firstDirtyOffset,
math.max(firstDirtyOffset, firstUncleanableOffset)).map(_.size.toLong).sum
-
-(firstUncleanableOffset, cleanableBytes)
- }
}
/**
@@ -1048,7 +1037,7 @@ private case class LogToClean(topicPartition:
TopicPartition,
uncleanableOffset: Long,
needCompactionNow: Boolean = false) extends
Ordered[LogToClean] {
val cleanBytes = log.logSegments(-1, firstDirtyOffset).map(_.size.toLong).sum
- val (firstUncleanableOffset, cleanableBytes) =
LogCleaner.calculateCleanableBytes(log, firstDirtyOffset, uncleanableOffset)
+ val (firstUncleanableOffset, cleanableBytes) =
LogCleanerManager.calculateCleanableBytes(log, firstDirtyOffset,
uncleanableOffset)
val totalBytes = cleanBytes + cleanableBytes
val cleanableRatio = cleanableBytes / totalBytes.toDouble
override def compare(that: LogToClean): Int =
math.signum(this.cleanableRatio - that.cleanableRatio).toInt
diff --git a/core/src/main/scala/kafka/log/LogCleanerManager.scala
b/core/src/main/scala/kafka/log/LogCleanerManager.scala
index efa4d94..18bb871 100755
--- a/core/src/main/scala/kafka/log/LogCleanerManager.scala
+++
[kafka] branch trunk updated (dbce4c1 -> 1cd5d0d)
This is an automated email from the ASF dual-hosted git repository. mjsax pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/kafka.git. from dbce4c1 MINOR: Update docs about ZooKeeper upgrade issue from 3.4.X to 3.5.6 add 1cd5d0d MINOR: clarify statefulness in transform/process variations (#7668) No new revisions were added by this update. Summary of changes: .../org/apache/kafka/streams/kstream/KStream.java | 69 +- 1 file changed, 53 insertions(+), 16 deletions(-)
[kafka] branch trunk updated (1cd5d0d -> 38ed692)
This is an automated email from the ASF dual-hosted git repository. mjsax pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/kafka.git. from 1cd5d0d MINOR: clarify statefulness in transform/process variations (#7668) add 38ed692 MINOR: clarify KafkaStreams.close javadoc (#7605) No new revisions were added by this update. Summary of changes: streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
