[GitHub] [beam] youngoli commented on pull request #12350: [BEAM-10289] Dynamic splitting implementation.
youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-668301428 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12350: [BEAM-10289] Dynamic splitting implementation.
youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-667466351 Added a new commit addressing the latest comments, and also adding a timeout to the usage of the SplittableUnit channel in DataSource, to avoid the possibility of it blocking for arbitrarily long periods of time. The timeout I chose was arbitrary, and the idea of choosing a timeout in the first place instead of just a default case is to avoid the subtle edge case where a split occurs right after the current element has incremented in DataSource, but before that element has begun processing in the splittable unit. Without the timeout, those cases will result in avoiding a sub-element split when one was possible if we just waited slightly longer. But maybe I'm just overthinking it, and that case is rare enough that it's not worth worrying about. I'm still pretty undecided on it, and I'd like to hear your opinion @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12350: [BEAM-10289] Dynamic splitting implementation.
youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-666769902 Good idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12350: [BEAM-10289] Dynamic splitting implementation.
youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-664823531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12350: [BEAM-10289] Dynamic splitting implementation.
youngoli commented on pull request #12350: URL: https://github.com/apache/beam/pull/12350#issuecomment-662810997 CC: @lostluck Doesn't need a review yet, but if you want to give it a look, have at it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
