damccorm commented on code in PR #36346:
URL: https://github.com/apache/beam/pull/36346#discussion_r2409081910
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/DataflowGroupByKey.java:
##########
@@ -171,10 +208,22 @@ public String getUrn() {
return PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN;
}
+ @Override
+ public String getUrn(DataflowGroupByKey<?, ?> transform) {
+ if (transform.surroundsGBEK()) {
+ return "beam:transform:group_by_key_wrapper:v1";
Review Comment:
Sure, done
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/DataflowGroupByKey.java:
##########
@@ -171,10 +208,22 @@ public String getUrn() {
return PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN;
}
+ @Override
+ public String getUrn(DataflowGroupByKey<?, ?> transform) {
+ if (transform.surroundsGBEK()) {
+ return "beam:transform:group_by_key_wrapper:v1";
+ }
+ return PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN;
+ }
+
@Override
@SuppressWarnings("nullness")
public RunnerApi.FunctionSpec translate(
AppliedPTransform<?, ?, DataflowGroupByKey<?, ?>> transform,
SdkComponents components) {
+ if (transform.getTransform().surroundsGBEK()) {
Review Comment:
Sure, made this nullable
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java:
##########
@@ -413,6 +414,40 @@ public Long create(PipelineOptions options) {
void setUserAgent(String userAgent);
+ /**
+ * A string defining whether GroupByKey transforms should be replaced by
GroupByEncryptedKey
+ *
+ * <p>Beam will infer the secret type and value based on the secret itself.
This guarantees that
+ * any data at rest during the performing a GBK, so this can be used to
guarantee that data is not
+ * unencrypted. Runners with this behavior include the Dataflow, Flink, and
Spark runners. The
+ * option should be structured like:
+ *
+ * <pre><code>
+ * --gbek=type:<secret_type>;<secret_param>:<value>
+ * </code></pre>
+ *
+ * for example:
+ *
+ * <pre><code>
+ * --gbek=type:GcpSecret;version_name:my_secret/versions/latest
+ * </code></pre>
+ *
+ * All variables should use snake case to allow consistency across languages.
Review Comment:
I noticed this a minute ago writing a xlang test - updated to avoid this,
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
