rhauch commented on a change in pull request #8858:
URL: https://github.com/apache/kafka/pull/8858#discussion_r439467273
##########
File path: docs/connect.html
##########
@@ -258,6 +258,29 @@ <h4><a id="connect_rest" href="#connect_rest">REST
API</a></h4>
<li><code>GET /</code>- return basic information about the Kafka
Connect cluster such as the version of the Connect worker that serves the REST
request (including git commit ID of the source code) and the Kafka cluster ID
that is connected to.
</ul>
+ <h4><a id="connect_errorreporting" href="#connect_errorreporting">Error
Reporting in Connect</a></h4>
+
+ <p>Kafka Connect provides error reporting for errors encountered along
various stages of processing in Kafka Connect. A connector can be configured to
enable log reporting, which logs the error context along with the standard
application logs, as well as to enable dead letter queue reporting (within sink
connectors only), which will write the original record (from the Kafka topic
the sink connector is consuming from) into a configurable Kafka topic.
Configuring these reporters will enable for automatic error reporting along
both the conversion and transformation stages of Connect. For error reporting
for records after they reach the connector, the <code>Errant Record
Reporter</code> must be implemented within the specific connector along with
the configuration.</p>
+
+ <h5><a id="connect_errantrecordreporting"
href="#connect-errantrecordreporting">Errant Record Reporter</a></h5>
+
+ <p>The <code>ErrantRecordReporter</code> can be initialized within the
<code>SinkTask</code> while preserving backwards compatibility with the
following example:</p>
+
+ <pre class="brush: java;">
+ private ErrantRecordReporter reporter;
+
+ @Override
+ public void start(Map<String, String> props) {
+ ...
+ try {
+ reporter = context.failedRecordReporter(); // may be null if
DLQ not enabled
+ } catch (NoSuchMethodException | NoClassDefFoundError e) {
+ // Will occur in Connect runtimes earlier than 2.6
+ reporter = null;
+ }
+ }
+ </pre>
+
Review comment:
Let's add a new section just before "Resuming from Previous Offsets"
that talks about how sink tasks can use the error reporter, similar to the
["Example Usage" in the
KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors#KIP610:ErrorReportinginSinkConnectors-ExampleUsage),
but don't show using the returned future, since the guarantees provided by the
framework about completing the writing of the errors are likely good enough for
*most* connectors, and developers of any more sophisticated connector that
needs stronger guarantees can read the JavaDoc.
##########
File path: docs/connect.html
##########
@@ -258,6 +258,29 @@ <h4><a id="connect_rest" href="#connect_rest">REST
API</a></h4>
<li><code>GET /</code>- return basic information about the Kafka
Connect cluster such as the version of the Connect worker that serves the REST
request (including git commit ID of the source code) and the Kafka cluster ID
that is connected to.
</ul>
+ <h4><a id="connect_errorreporting" href="#connect_errorreporting">Error
Reporting in Connect</a></h4>
+
+ <p>Kafka Connect provides error reporting for errors encountered along
various stages of processing in Kafka Connect. A connector can be configured to
enable log reporting, which logs the error context along with the standard
application logs, as well as to enable dead letter queue reporting (within sink
connectors only), which will write the original record (from the Kafka topic
the sink connector is consuming from) into a configurable Kafka topic.
Configuring these reporters will enable for automatic error reporting along
both the conversion and transformation stages of Connect. For error reporting
for records after they reach the connector, the <code>Errant Record
Reporter</code> must be implemented within the specific connector along with
the configuration.</p>
Review comment:
A few suggestions for readability and to add some information that could
have been added with
[KIP-298](https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect):
```suggestion
<p>Kafka Connect provides error reporting to handle errors encountered
along various stages of processing. By default, any error encountered during
conversion or within transformations will cause the connector to fail. Each
connector configuration can also enable tolerating such errors by skipping
them, optionally writing each error and the details of the failed operation and
problematic record (with various levels of detail) to the Connect application
log. These mechanisms also capture errors when a sink connector is processing
the messages consumed from its Kafka topics, and all of the errors can be
written to a configurable "dead letter queue" (DLQ) Kafka topic.</p>
<p>To report errors to the log and/or DLQ, set
<code>errors.tolerance=all</code> in the connector configuration. Likewise, set
<code>errors.log.enable=true</code> in the connector configuration to log
details of each error and problem record's topic, partition, and offset. For
additional debugging purposes, set
<code>errors.log.include.messages=true</code> to also log the problem record
key, value, and headers to the log (note this may log sensitive information).
<p>To report errors within a sink connector's converter, transforms, or
in the connector itself to a dead letter queue topic, set
<code>errors.deadletterqueue.topic.name</code>, and optionally
<code>errors.deadletterqueue.context.headers.enable=true</code>.</p>
<p>For example, a connector will fail immediately upon an error or
exception. Although it is not necessary to add extra configuration properties
for this behavior, adding the following properties to a sink connector
configuration would achieve this "fail fast" behavior:</p>
<pre class="brush: text;">
# disable retries on failure
errors.retry.timeout=0
# do not log the error and their contexts
errors.log.enable=false
# do not record errors in a dead letter queue topic
errors.deadletterqueue.topic.name=
# Fail on first error
errors.tolerance=none
</pre>
<p>The following configuration shows how to setup error handling with
multiple retries, logging both to the application logs and a Kafka topic with
infinite tolerance:</p>
<pre class="brush: text;">
# retry for at most 10 minutes times waiting up to 30 seconds
between consecutive failures
errors.retry.timeout=600000
errors.retry.delay.max.ms=30000
# log error context along with application logs, but do not include
configs and messages
errors.log.enable=true
errors.log.include.messages=false
# produce error context into the Kafka topic
errors.deadletterqueue.topic.name=my-connector-errors
# Tolerate all errors.
errors.tolerance=all
</pre>
```
@aakashnshah, @wicknicks: what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
