afedulov commented on PR #23777: URL: https://github.com/apache/flink/pull/23777#issuecomment-1824683968
There are some things that can probably be done, but not in the scope of this PR. You can see how much complexity of the new Source API is already hidden behind the `IteratorSourceReaderBase`, the `DataGeneratorSource` and the `GeneratorFunction`. We are getting there, but in order to come up with meaningful abstractions we need to learn more about common patterns and we do so during migration of the existing SourceFunctions. I do not think there is an easy way to some up with something completely generic because of how the new Source imposes implementation of all of the low-level details with very specific classes (Splits, SplitEnumerators, serializers). See, for instance, what is going on in the `NumberSequenceSource` that we wrap around in the `DataGeneratorSource` that this implementation relies on: https://github.com/apache/flink/blob/e4b000818c15b5b781c4e5262ba83bfc9d65121a/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/NumberSequenceSource.java#L218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
