kennknowles opened a new issue, #18406: URL: https://github.com/apache/beam/issues/18406
AvroReader is able to detect when it's in the last block: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L728 It could also use this information to avoid wastefully producing dynamic splits starting in the range of the current block. One way to do this would be to have OffsetRangeTracker have a "claim range" operation: claim range of [a, b) is, in terms of correctness, equivalent to claiming "a" (it checks whether "a" is within the range), but sets the last claimed position to "b" rather than "a", thus protecting more positions from being split away. Imported from Jira [BEAM-2716](https://issues.apache.org/jira/browse/BEAM-2716). Original Jira may contain additional context. Reported by: jkff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
