[kafka] branch 2.2 updated: KAFKA-7884; Docs for message.format.version should display valid values (#6209)
This is an automated email from the ASF dual-hosted git repository.
jgus pushed a commit to branch 2.2
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/2.2 by this push:
new 76cac52 KAFKA-7884; Docs for message.format.version should display
valid values (#6209)
76cac52 is described below
commit 76cac5299d01e8155157e6c2024c62cf1977570f
Author: Lee Dongjin
AuthorDate: Sat Feb 16 09:50:01 2019 +0900
KAFKA-7884; Docs for message.format.version should display valid values
(#6209)
The config docs for message.format.version and log.message.format.version
show invalid (corrupt?) "valid values". The problem is
that`ApiVersionValidator#toString` is missing. In contrast, all other
Validators like `ThrottledReplicaListValidator` or `Range`, have its own
`toString` method. This patch solves this problem by adding
`ApiVersionValidator#toString`. It also provides a unit test for it.
Reviewers: Jason Gustafson
---
core/src/main/scala/kafka/api/ApiVersion.scala | 2 ++
core/src/test/scala/unit/kafka/api/ApiVersionTest.scala | 7 +++
2 files changed, 9 insertions(+)
diff --git a/core/src/main/scala/kafka/api/ApiVersion.scala
b/core/src/main/scala/kafka/api/ApiVersion.scala
index a68bcf0..8f78a96 100644
--- a/core/src/main/scala/kafka/api/ApiVersion.scala
+++ b/core/src/main/scala/kafka/api/ApiVersion.scala
@@ -307,4 +307,6 @@ object ApiVersionValidator extends Validator {
case e: IllegalArgumentException => throw new ConfigException(name,
value.toString, e.getMessage)
}
}
+
+ override def toString: String = "[" +
ApiVersion.allVersions.map(_.version).distinct.mkString(", ") + "]"
}
diff --git a/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
b/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
index f3e4294..571a077 100644
--- a/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
+++ b/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
@@ -119,4 +119,11 @@ class ApiVersionTest {
assertEquals("0.11.0", KAFKA_0_11_0_IV0.shortVersion)
}
+ @Test
+ def testApiVersionValidator(): Unit = {
+val str = ApiVersionValidator.toString
+val apiVersions = str.slice(1, str.length).split(",")
+assertEquals(ApiVersion.allVersions.size, apiVersions.length)
+ }
+
}
[kafka] branch trunk updated: KAFKA-7884; Docs for message.format.version should display valid values (#6209)
This is an automated email from the ASF dual-hosted git repository.
jgus pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 44257f2 KAFKA-7884; Docs for message.format.version should display
valid values (#6209)
44257f2 is described below
commit 44257f293732588fc493f072b0cee4bd3a1840ab
Author: Lee Dongjin
AuthorDate: Sat Feb 16 09:50:01 2019 +0900
KAFKA-7884; Docs for message.format.version should display valid values
(#6209)
The config docs for message.format.version and log.message.format.version
show invalid (corrupt?) "valid values". The problem is
that`ApiVersionValidator#toString` is missing. In contrast, all other
Validators like `ThrottledReplicaListValidator` or `Range`, have its own
`toString` method. This patch solves this problem by adding
`ApiVersionValidator#toString`. It also provides a unit test for it.
Reviewers: Jason Gustafson
---
core/src/main/scala/kafka/api/ApiVersion.scala | 2 ++
core/src/test/scala/unit/kafka/api/ApiVersionTest.scala | 7 +++
2 files changed, 9 insertions(+)
diff --git a/core/src/main/scala/kafka/api/ApiVersion.scala
b/core/src/main/scala/kafka/api/ApiVersion.scala
index a68bcf0..8f78a96 100644
--- a/core/src/main/scala/kafka/api/ApiVersion.scala
+++ b/core/src/main/scala/kafka/api/ApiVersion.scala
@@ -307,4 +307,6 @@ object ApiVersionValidator extends Validator {
case e: IllegalArgumentException => throw new ConfigException(name,
value.toString, e.getMessage)
}
}
+
+ override def toString: String = "[" +
ApiVersion.allVersions.map(_.version).distinct.mkString(", ") + "]"
}
diff --git a/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
b/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
index f3e4294..571a077 100644
--- a/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
+++ b/core/src/test/scala/unit/kafka/api/ApiVersionTest.scala
@@ -119,4 +119,11 @@ class ApiVersionTest {
assertEquals("0.11.0", KAFKA_0_11_0_IV0.shortVersion)
}
+ @Test
+ def testApiVersionValidator(): Unit = {
+val str = ApiVersionValidator.toString
+val apiVersions = str.slice(1, str.length).split(",")
+assertEquals(ApiVersion.allVersions.size, apiVersions.length)
+ }
+
}
[kafka] branch trunk updated: MINOR: Save failed test output to build output directory
This is an automated email from the ASF dual-hosted git repository.
ewencp pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new ed30712 MINOR: Save failed test output to build output directory
ed30712 is described below
commit ed3071231aee1ba8a5c2c496112dd6034f9bf942
Author: Ewen Cheslack-Postava
AuthorDate: Fri Feb 15 10:50:08 2019 -0800
MINOR: Save failed test output to build output directory
Author: Ewen Cheslack-Postava
Reviewers: Colin Patrick McCabe
Closes #6234 from ewencp/test-logs
---
build.gradle | 66 +++-
1 file changed, 65 insertions(+), 1 deletion(-)
diff --git a/build.gradle b/build.gradle
index 420edf7..ff316c8 100644
--- a/build.gradle
+++ b/build.gradle
@@ -15,6 +15,8 @@
import org.ajoberstar.grgit.Grgit
+import java.nio.charset.StandardCharsets
+
buildscript {
repositories {
mavenCentral()
@@ -139,6 +141,7 @@ if (file('.git').exists()) {
}
}
+
subprojects {
apply plugin: 'java'
// apply the eclipse plugin only to subprojects that hold code. 'connect' is
just a folder.
@@ -204,6 +207,65 @@ subprojects {
def testLoggingEvents = ["passed", "skipped", "failed"]
def testShowStandardStreams = false
def testExceptionFormat = 'full'
+ // Gradle built-in logging only supports sending test output to stdout,
which generates a lot
+ // of noise, especially for passing tests. We really only want output for
failed tests. This
+ // hooks into the output and logs it (so we don't have to buffer it all in
memory) and only
+ // saves the output for failing tests. Directory and filenames are such that
you can, e.g.,
+ // create a Jenkins rule to collect failed test output.
+ def logTestStdout = {
+def testId = { TestDescriptor descriptor ->
+ "${descriptor.className}.${descriptor.name}".toString()
+}
+
+def logFiles = new HashMap()
+def logStreams = new HashMap()
+beforeTest { TestDescriptor td ->
+ def tid = testId(td)
+ def logFile = new File(
+ "${projectDir}/build/reports/testOutput/${tid}.test.stdout")
+ logFile.parentFile.mkdirs()
+ logFiles.put(tid, logFile)
+ logStreams.put(tid, new FileOutputStream(logFile))
+}
+onOutput { TestDescriptor td, TestOutputEvent toe ->
+ def tid = testId(td)
+ // Some output can happen outside the context of a specific test (e.g.
at the class level)
+ // and beforeTest/afterTest seems to not be invoked for these cases (and
similarly, there's
+ // a TestDescriptor hierarchy that includes the thread executing the
test, Gradle tasks,
+ // etc). We see some of these in practice and it seems like something
buggy in the Gradle
+ // test runner since we see it *before* any tests and it is frequently
not related to any
+ // code in the test (best guess is that it is tail output from last
test). We won't have
+ // an output file for these, so simply ignore them. If they become
critical for debugging,
+ // they can be seen with showStandardStreams.
+ if (td.name == td.className) {
+return
+ }
+ try {
+logStreams.get(tid).write(toe.message.getBytes(StandardCharsets.UTF_8))
+ } catch (Exception e) {
+println "ERROR: Failed to write output for test ${tid}"
+e.printStackTrace()
+ }
+}
+afterTest { TestDescriptor td, TestResult tr ->
+ def tid = testId(td)
+ try {
+logStreams.get(tid).close()
+if (tr.resultType != TestResult.ResultType.FAILURE) {
+ logFiles.get(tid).delete()
+} else {
+ def file = logFiles.get(tid)
+ println "${tid} failed, log available in ${file}"
+}
+ } catch (Exception e) {
+println "ERROR: Failed to close stdout file for ${tid}"
+e.printStackTrace()
+ } finally {
+logFiles.remove(tid)
+logStreams.remove(tid)
+ }
+}
+ }
test {
maxParallelForks = userMaxForks ?: Runtime.runtime.availableProcessors()
@@ -216,7 +278,7 @@ subprojects {
showStandardStreams = userShowStandardStreams ?: testShowStandardStreams
exceptionFormat = testExceptionFormat
}
-
+logTestStdout.rehydrate(delegate, owner, this)()
}
task integrationTest(type: Test, dependsOn: compileJava) {
@@ -230,6 +292,7 @@ subprojects {
showStandardStreams = userShowStandardStreams ?: testShowStandardStreams
exceptionFormat = testExceptionFormat
}
+logTestStdout.rehydrate(delegate, owner, this)()
useJUnit {
includeCategories 'org.apache.kafka.test.IntegrationTest'
@@ -248,6 +311,7 @@ subprojects {
showStandardStreams = userShowStandardStreams ?: testShowStandardStreams
exceptionFormat = testExceptionFormat
}
+logTestStdout.rehydrate(delegate, owner, this)()
[kafka] branch trunk updated: MINOR: add test for StreamsSmokeTestDriver (#6231)
This is an automated email from the ASF dual-hosted git repository.
bbejeck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3656ad9 MINOR: add test for StreamsSmokeTestDriver (#6231)
3656ad9 is described below
commit 3656ad93bc7fb74138e36d39575c3b86c319577b
Author: John Roesler
AuthorDate: Fri Feb 15 10:23:57 2019 -0600
MINOR: add test for StreamsSmokeTestDriver (#6231)
* MINOR: add test for StreamsSmokeTestDriver
Hi @bbejeck @mjsax @ableegoldman @guozhangwang ,
Please take a look at this when you get the chance.
The primary concern is adding the test. It will help us verify changes to
the smoke test (such as adding suppression).
I've also added some extra output to the smoke test stdout, which will
hopefully aid us in diagnosing the flaky tests.
Finally, I bundled in some cleanup. It was my intention to do that in a
separate PR, but it wound up getting smashed together during refactoring.
Please let me know if you'd prefer for me to pull any of these out into a
separate request.
Thanks,
-John
Also, add more output for debuggability
* cleanup
* cleanup
* refactor
* refactor
* remove redundant printlns
* Update EmbeddedKafkaCluster.java
* move to integration package
* replace early-exit on pass
* use classrule for embedded kafka
* pull in smoke test improvements from side branch
* try-with-resources
* format events instead of printing long lines
* minor formatting fix
Reviewers: Matthias J. Sax , Bill Bejeck
---
.../SmokeTestDriverIntegrationTest.java| 134 +
.../kafka/streams/tests/SmokeTestClient.java | 189 +++
.../kafka/streams/tests/SmokeTestDriver.java | 564 -
.../apache/kafka/streams/tests/SmokeTestUtil.java | 7 +-
.../kafka/streams/tests/StreamsSmokeTest.java | 10 +-
5 files changed, 438 insertions(+), 466 deletions(-)
diff --git
a/streams/src/test/java/org/apache/kafka/streams/integration/SmokeTestDriverIntegrationTest.java
b/streams/src/test/java/org/apache/kafka/streams/integration/SmokeTestDriverIntegrationTest.java
new file mode 100644
index 000..82f86c2
--- /dev/null
+++
b/streams/src/test/java/org/apache/kafka/streams/integration/SmokeTestDriverIntegrationTest.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.tests.SmokeTestClient;
+import org.apache.kafka.streams.tests.SmokeTestDriver;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+import static org.apache.kafka.streams.tests.SmokeTestDriver.generate;
+import static org.apache.kafka.streams.tests.SmokeTestDriver.verify;
+
+public class SmokeTestDriverIntegrationTest {
+@ClassRule
+public static final EmbeddedKafkaCluster CLUSTER = new
EmbeddedKafkaCluster(3);
+
+
+private static class Driver extends Thread {
+private String bootstrapServers;
+private int numKeys;
+private int maxRecordsPerKey;
+private Exception exception = null;
+private SmokeTestDriver.VerificationResult result;
+
+private Driver(final String bootstrapServers, final int numKeys, final
int maxRecordsPerKey) {
+this.bootstrapServers = bootstrapServers;
+this.numKeys = numKeys;
+this.maxRecordsPerKey = maxRecordsPerKey;
+}
+
+@Override
+public void run() {
+try {
+final Map> allData =
generate(bootstrapServers, numKeys, maxRecordsPerKey, true);
+result = verify(bootstrapServers, allData, maxRecordsPerKey);
+
+} catch (final Exception ex) {
+this.exception = ex;
[kafka] 02/02: Update versions to 2.1.2-SNAPSHOT
This is an automated email from the ASF dual-hosted git repository.
cmccabe pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 9729a34f5050d6365ab0493bb7356a8757e97818
Author: Colin P. Mccabe
AuthorDate: Fri Feb 15 01:12:44 2019 -0800
Update versions to 2.1.2-SNAPSHOT
---
docs/js/templateData.js| 2 +-
gradle.properties | 2 +-
kafka-merge-pr.py | 2 +-
streams/quickstart/java/pom.xml| 2 +-
streams/quickstart/java/src/main/resources/archetype-resources/pom.xml | 2 +-
streams/quickstart/pom.xml | 2 +-
tests/kafkatest/__init__.py| 2 +-
tests/kafkatest/version.py | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/docs/js/templateData.js b/docs/js/templateData.js
index f65e2af..c457546 100644
--- a/docs/js/templateData.js
+++ b/docs/js/templateData.js
@@ -19,6 +19,6 @@ limitations under the License.
var context={
"version": "21",
"dotVersion": "2.1",
-"fullDotVersion": "2.1.1",
+"fullDotVersion": "2.1.2",
"scalaVersion": "2.11"
};
diff --git a/gradle.properties b/gradle.properties
index 2abe9d1..a97dc7b 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -19,7 +19,7 @@ group=org.apache.kafka
# - tests/kafkatest/__init__.py,
# - tests/kafkatest/version.py (variable DEV_VERSION)
# - kafka-merge-pr.py.
-version=2.1.1
+version=2.1.2-SNAPSHOT
scalaVersion=2.11.12
task=build
org.gradle.jvmargs=-Xmx1024m -Xss2m
diff --git a/kafka-merge-pr.py b/kafka-merge-pr.py
index 925b646..00e8f72 100755
--- a/kafka-merge-pr.py
+++ b/kafka-merge-pr.py
@@ -70,7 +70,7 @@ TEMP_BRANCH_PREFIX = "PR_TOOL"
DEV_BRANCH_NAME = "trunk"
-DEFAULT_FIX_VERSION = os.environ.get("DEFAULT_FIX_VERSION", "2.1.1")
+DEFAULT_FIX_VERSION = os.environ.get("DEFAULT_FIX_VERSION", "2.1.2")
def get_json(url):
try:
diff --git a/streams/quickstart/java/pom.xml b/streams/quickstart/java/pom.xml
index 44754e6..7c58b84 100644
--- a/streams/quickstart/java/pom.xml
+++ b/streams/quickstart/java/pom.xml
@@ -26,7 +26,7 @@
org.apache.kafka
streams-quickstart
-2.1.1
+2.1.2-SNAPSHOT
..
diff --git
a/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
b/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
index 37a7459..8c8c655 100644
--- a/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
+++ b/streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
@@ -29,7 +29,7 @@
UTF-8
-2.1.1
+2.1.2-SNAPSHOT
1.7.7
1.2.17
diff --git a/streams/quickstart/pom.xml b/streams/quickstart/pom.xml
index 7c2d48f..b6ee257 100644
--- a/streams/quickstart/pom.xml
+++ b/streams/quickstart/pom.xml
@@ -22,7 +22,7 @@
org.apache.kafka
streams-quickstart
pom
-2.1.1
+2.1.2-SNAPSHOT
Kafka Streams :: Quickstart
diff --git a/tests/kafkatest/__init__.py b/tests/kafkatest/__init__.py
index c729d56..257ef1e 100644
--- a/tests/kafkatest/__init__.py
+++ b/tests/kafkatest/__init__.py
@@ -22,4 +22,4 @@
# Instead, in
#
# For example, when Kafka is at version 1.0.0-SNAPSHOT, this should be
something like "1.0.0
-__version__ = '2.1.1'
+__version__ = '2.1.2.dev0'
diff --git a/tests/kafkatest/version.py b/tests/kafkatest/version.py
index 629dec0..d880ebb 100644
--- a/tests/kafkatest/version.py
+++ b/tests/kafkatest/version.py
@@ -60,7 +60,7 @@ def get_version(node=None):
return DEV_BRANCH
DEV_BRANCH = KafkaVersion("dev")
-DEV_VERSION = KafkaVersion("2.1.1-SNAPSHOT")
+DEV_VERSION = KafkaVersion("2.1.2-SNAPSHOT")
# 0.8.2.x versions
V_0_8_2_1 = KafkaVersion("0.8.2.1")
[kafka] branch 2.1 updated (f15678e -> 9729a34)
This is an automated email from the ASF dual-hosted git repository. cmccabe pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/kafka.git. from f15678e MINOR: Make MockClient#poll() more thread-safe (#5942) add 21234be Bump version to 2.1.1 new d69e03e Merge tag '2.1.1' into 2.1 new 9729a34 Update versions to 2.1.2-SNAPSHOT The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/js/templateData.js| 2 +- gradle.properties | 2 +- kafka-merge-pr.py | 2 +- streams/quickstart/java/pom.xml| 2 +- .../java/src/main/resources/archetype-resources/pom.xml| 2 +- streams/quickstart/pom.xml | 2 +- tests/kafkatest/__init__.py| 10 +- tests/kafkatest/version.py | 2 +- 8 files changed, 12 insertions(+), 12 deletions(-)
[kafka] 01/02: Merge tag '2.1.1' into 2.1
This is an automated email from the ASF dual-hosted git repository. cmccabe pushed a commit to branch 2.1 in repository https://gitbox.apache.org/repos/asf/kafka.git commit d69e03e176f49cee91a0322afcda816a6984543e Merge: f15678e 21234be Author: Colin P. Mccabe AuthorDate: Fri Feb 15 01:08:17 2019 -0800 Merge tag '2.1.1' into 2.1 Apache Kafka 2.1.1 release gradle.properties | 2 +- streams/quickstart/java/pom.xml| 2 +- .../java/src/main/resources/archetype-resources/pom.xml| 2 +- streams/quickstart/pom.xml | 2 +- tests/kafkatest/__init__.py| 10 +- 5 files changed, 9 insertions(+), 9 deletions(-)
[kafka] annotated tag 2.1.1 created (now a29fd84)
This is an automated email from the ASF dual-hosted git repository. cmccabe pushed a change to annotated tag 2.1.1 in repository https://gitbox.apache.org/repos/asf/kafka.git. at a29fd84 (tag) tagging 21234bee31165527859b46ea48c46b76532f7a37 (commit) replaces 2.1.0 by Colin P. Mccabe on Fri Feb 15 01:04:52 2019 -0800 - Log - Apache Kafka 2.1.1 release --- No new revisions were added by this update.
