dejii commented on PR #38149: URL: https://github.com/apache/beam/pull/38149#issuecomment-4239348212
Hey @stankiewicz - thanks for the review. > Doing it per bundle is an overkill, especially if for each bundle there are many tables to write to, catalog will waste plenty of time on loading table credentials. Just to clarify - closing `FileIO` per bundle is the current behavior, and it's not wasted time but thrown errors when using dynamic destinations: the catalog tries to reuse a dead connection pool on every subsequent bundle, and the pipeline ultimately fails. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
