youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-667466351
Added a new commit addressing the latest comments, and also adding a timeout to the usage of the SplittableUnit channel in DataSource, to avoid the possibility of it blocking for arbitrarily long periods of time. The timeout I chose was arbitrary, and the idea of choosing a timeout in the first place instead of just a default case is to avoid the subtle edge case where a split occurs right after the current element has incremented in DataSource, but before that element has begun processing in the splittable unit. Without the timeout, those cases will result in avoiding a sub-element split when one was possible if we just waited slightly longer. But maybe I'm just overthinking it, and that case is rare enough that it's not worth worrying about. I'm still pretty undecided on it, and I'd like to hear your opinion @lostluck ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
