awadhoot-sq opened a new issue, #29515: URL: https://github.com/apache/beam/issues/29515
### What happened? I am running Pipeline with WriteToFiles the destination is fetched from record and then files are written to GCP cloud Storage we found out that few of the records are missing in the files, and during debug we found out few files are left in the temp directory beam logs this while running locally but not in Dataflow pipelines the temp directory is deleted when running on dataflow. but its retained when running locally. ``` INFO:apache_beam.io.fileio:Some files may be left orphaned in the temporary folder: ['data/temp\\1891039626663227874_034010af-241f-4930-a782-127098aba54b'] ``` we tried to play with "max_writers_per_bundle" and set it to 300, as numbers of files we are writing is between 100 - 300. there were fewer missing records than earlier but some are still missing (2 records missing for every 100 records) it does feel like a performance issue and there might be a need optimizing the parameter, but records missing without any errors or warning is cause of a concern for us. ``` SDK version Apache Beam Python 3.11 SDK 2.50.0 machineType n2-standard-4 ``` ### Issue Priority Priority: 1 (data loss / total loss of function) ### Issue Components - [X] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [X] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
