lostluck commented on issue #32498: URL: https://github.com/apache/beam/issues/32498#issuecomment-2361220722
This is an over splitting problem with the prism runner, due to higher latency with GCS. I had thought we sorted this out (there are some previous issues that were resolved previously but apparently not). This can be confirmed by comparing the behavior to a local file read vs from GCS. There's a bit of tension between certain goals of the Prism runner (fast execution in test situations) and practical use (reading from remote stores) that the current split policy doesn't satisfy. That needs to be fixed. The solution here is that we make the split policy more configurable so we can get the desired fast behavior check for the splitting tests, but increase the default wait time so the example works in higher latency environments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
