damccorm commented on code in PR #31217:
URL: https://github.com/apache/beam/pull/31217#discussion_r1595652693
##########
settings.gradle.kts:
##########
@@ -121,6 +121,13 @@ include(":runners:core-java")
include(":runners:direct-java")
include(":runners:extensions-java:metrics")
/* Begin Flink Runner related settings */
+/* When updating these versions, please make sure that the following files are
updated as well:
+ * FLINK_VERSIONS in
.github/actions/setup-default-test-properties/test-properties.json
+ * flink_versions in sdks/go/examples/wasm/README.md
+ * PUBLISHED_FLINK_VERSIONS in
sdks/python/apache_beam/options/pipeline_options.py
+ * PUBLISHED_FLINK_VERSIONS in
sdks/typescript/src/apache_beam/runners/flink.ts
+ * verify versions in
website/www/site/content/en/documentation/runners/flink.md
Review Comment:
> On the other hand, can non-googler be confident enough to change these
settings? When can we change to 2.2? I really don't know, which is why it would
feel more 'stable' to offload this.
To be clear, I don't have any particular special insights here. I'd never
used Dataproc before I made my change and don't really have any connections to
the team. I chose 2.1 because that's what the rest of our project uses for
testing and I was trying to keep my changes scoped small for the bugfix, not
because there's a particular reason 2.2 wouldn't work. We could add some
comments recommending that approach if you'd like?
> I find it generally non-intuitive for OSS project to have such
dependencies on proprietary solutions.
I don't think this is particularly different than having a Dataflow runner
or supporting GCP/Azure/AWS filesystems (or other arbitrary IOs); there's no
reason we couldn't support other clouds or local setups (in fact we already do,
we just don't spin up a cluster automatically for those setups yet). The only
reason is that nobody from the community has had enough time or know-how to
contribute an implementation (which is also fine). I'd actually love it if we
had an AWS Managed Flink offering here, and we could very easily version that
differently than Dataproc.
The dataproc piece only impacts people who want to use dataproc and have
GCP-specific pipeline options set, which seems reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
