Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
github-actions[bot] commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-3831001083 This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
github-actions[bot] closed issue #31007: [Feature Request]: SpannerIO: support max commit delay URL: https://github.com/apache/beam/issues/31007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
github-actions[bot] commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-3705257498 This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
nielm commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-2066200195 Yes, setting commit timeouts on individual transactions is a relatively new feature in the spanner client libraries. However in Beam, you can only set the timeout on the entire Write transform, which is then the same as the existing RPC commit deadline parameter. In addition, there is a dependency here - the maximum commit delay will always be the RPC commit deadline. So while we appreciate the contribution, it is not necessary, as the existing parameter has the same effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
kberezin-nshl commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-2063319042 @nielm oh, I believe that is a completely different feature, this PR is about throughput optimized writes: [documentation link](https://cloud.google.com/spanner/docs/throughput-optimized-writes#set_mixed_commit_delay_times). It was only released on March 26th, a couple of weeks ago: [release notes link](https://cloud.google.com/spanner/docs/release-notes#March_26_2024) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
nielm commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-2063316855 Commit deadlines for writes have been supported since [2020](https://issues.apache.org/jira/browse/BEAM-9269) at the RPC level and can be supported in 3 ways: * using [Write.withCommitDeadline()](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L1184) * using [SpannerConfig.withCommitDeadline()](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java#L224) * using [SpannerConfig.withCommitRetrySettings()](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java#L255) The [default is 15seconds](https://github.com/apache/beam/blame/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java#L46C2-L48C35 ) with retry and backoff, because with very long commit deadlines, a pipeline can push spanner into an overload situation, and reduce overall throughput. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: SpannerIO: support max commit delay [beam]
kberezin-nshl commented on issue #31007: URL: https://github.com/apache/beam/issues/31007#issuecomment-2060420112 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
