DanielMorales9 commented on issue #32746: URL: https://github.com/apache/beam/issues/32746#issuecomment-2407342035
I updated the GCS-Connector to hadoop3-2.2.16 and I can see high latency warnings on all type of operations on the manifest files. <img width="1610" alt="Screenshot 2024-10-11 at 14 41 03" src="https://github.com/user-attachments/assets/57314dba-fb5e-428f-a914-e6f15c86c8be";> Also, the request count on GCS is quite high compared to 20RPS on Pubsub. <img width="1161" alt="Screenshot 2024-10-11 at 14 44 26" src="https://github.com/user-attachments/assets/94c4e434-e716-4a11-89cd-3fdca406ba39";> Data files are written but the consumer is slow on committing a snapshot and a new manifest list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
