damccorm opened a new issue, #20557: URL: https://github.com/apache/beam/issues/20557
In Apache Beam , while using below P transformation -*fileio.MatchAll() and fileio.ReadMatches()* dataflow job on load performance .( processing 48 million resources) For 2-3 minute Throughput(element/sec) goes to zero and no element was processed during that time , it comes back again back to average normal (0.25/sec) of my job In between these three minutes. There is no log from apache beam , if some IO error came or any warning. Please check for any fileIO known issue on latest version of *apache-beam to 2.23.0* Imported from Jira [BEAM-11015](https://issues.apache.org/jira/browse/BEAM-11015). Original Jira may contain additional context. Reported by: [email protected]. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
