kfaraz opened a new pull request, #16614:
URL: https://github.com/apache/druid/pull/16614
### Description
A streaming ingestion with multiple replicas may sometimes run into the
following error
when publishing segments to an interval. This can happen only when there is
a concurrent
REPLACE task has recently committed higher version segments to that same
interval.
```
java.util.concurrent.ExecutionException:
org.apache.druid.java.util.common.ISE: Failed to publish segments because of
[java.lang.RuntimeException: Inconsistency between stored metadata
state[KafkaDataSourceMetadata{xxx}]
and target state[KafkaDataSourceMetadata{yyy}]. Try resetting the
supervisor.]
```
This error does not cause any data loss but is an operational overhead since
it leads to unnecessary task failures.
The situation typically plays out as below:
- Streaming supervisor launches 2 replicas to append to an interval I
- Both replicas allocate and start appending to a segment S(v0,p1), i.e.
segment partition 1 on version 0.
- Concurrent REPLACE task commits a new version v1 in interval I, say
segments S(v1, p0), S(v1, p1)
- Version v0 is now completely overshadowed by version v1
- First replica publishes segment S(v0, p1), which is upgraded by the
overlord to S(v1, p2)
- Second replica tries to publish segment S(v0, p1) but fails because higher
offsets have already been committed
- Second replica then checks if its segments have already been published by
someone else
- In the absence of concurrent replace, the previous step would succeed thus
causing second replica to move ahead with the ingestion.
- But since concurrent replace has already overshadowed version v0, the
second replica is not able to find it in the set of "used and visible" segments
and thus it fails.
### Fix
While looking for published segments, the second replica should search not
only visible, but also overshadowed and unused segments.
### Changes
- Rename `UsedSegmentChecker` to `PublishedSegmentRetriever`
- Search all of visible, overshadowed and unused segments in
`ActionBasedPublishedSegmentRetriever`
- Add utility method `asAppenderatorMetadata()` in
`SegmentsAndCommitMetadata`
### To do
- Add more unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
