L1nq0 opened a new pull request, #9093:
URL: https://github.com/apache/storm/pull/9093
What this changes
When a tuple emitted by the kafka spout reaches the retry limit,
KafkaSpout.fail() acked it and logged the event at debug level. The event is
now logged at error level and states the consequence:
LOG.error("Reached maximum number of retries. Giving up on message [{}]:
the tuple will be acked and its offset and later "
+ "offsets may be committed even though the record was not
processed.", msgId);
The message id carries the topic, partition and offset, so the log line
identifies the record.
The same semantics are documented at the two places a user configures or
reacts to them. The KafkaSpoutRetryExponentialBackoff constructor javadoc now
states that a finite maxRetries also stops endless retries of tuples that fail
every time they are emitted, e.g. tuples the receiving worker drops because
they cannot be deserialized, and that such tuples are reported to
KafkaTupleListener.onMaxRetryReached. The onMaxRetryReached javadoc now states
that the tuple is acked right after the callback, that commits can then move
past its offset, and that the callback is the last point at which the record
can be retained, with the msgId identifying the record by topic, partition and
offset.
Motivation
Issue #9078: with the default retry service, a record whose tuples always
fail keeps the spout from committing any offset past it, and the only signal
that the retry limit was reached was a debug-level line that most deployments
never see. Raising the give-up event to error level makes the give-up visible
to operators exactly where the guarantee is lost.
Scope
The retry mechanism, the ack and commit behavior and onMaxRetryReached
itself are unchanged. Questions on how give-up records should be retained by
default, e.g. dead letter handling or a listener capture example in the docs,
are left open in the issue. The storm-kafka-client test suite passes with no
changes (107/107); no existing test asserts the old log wording.
Relates to #9078
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
