rizenfrmtheashes commented on issue #23670: URL: https://github.com/apache/beam/issues/23670#issuecomment-1283048166
I think it only happens with dynamic destinations because there aren't any remove table steps in static destinations. That basically directly loads the data into destination table, instead of using a temp table intermediate step (there is some stuff around if some data hits some number thresholds to require partitioning that falls outside this, but that's an edge case). I just checked the job I use in production where this occurs, and the last autoscaling event occurred a little less than 2 hours prior to the first log where this error gets reported. So it's likely not autoscaling. I didn't see a one off instance re-allocation either during that time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
