boyuanzz commented on pull request #14811:
URL: https://github.com/apache/beam/pull/14811#issuecomment-877442652
Double checked the `SpannerIO` implementation. It seems like the current
implementation is **not** built on top of `BoundedSource`. The read is
expanded into: Create() -> GeneratePartitions -> Reshuffle -> DoFn reading from
a partition. And it seems like we are not able to split inside one
partition(I'm not 100% sure whether it's true).
So the value of converting this read into `SplittableDoFn` is:
* Taking advantage of expansion of SplittableDoFn. That means we can save
the expansion of read to `Create() -> SplittableDoFn()`. But it's not fully
true before resolving https://issues.apache.org/jira/browse/BEAM-12522.
@chamikaramj Do you think it's still valuable to convert SpannerIO into
SplittalbeDoFn? Or do I miss anything there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
