y1chi commented on a change in pull request #15137:
URL: https://github.com/apache/beam/pull/15137#discussion_r672643265
##########
File path:
model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
##########
@@ -440,3 +441,25 @@ examples:
shardId: "",
key: "key"
}
+
+---
+
+# Java code snippet to generate example bytes:
+# TimestampPrefixingWindowCoder<IntervalWindow> coder =
TimestampPrefixingWindowCoder.of(IntervalWindowCoder.of());
+# Instant end = new Instant(-9223372036854410L);
+# Duration span = Duration.millis(365L);
+# IntervalWindow window = new IntervalWindow(end.minus(span), span);
+# byte[] bytes = CoderUtils.encodeToByteArray(coder, window);
+# String str = new String(bytes,
java.nio.charset.StandardCharsets.ISO_8859_1);
+# String example = "";
+# for(int i = 0; i < str.length(); i++){
+# example += CharUtils.unicodeEscaped(str.charAt(i));
+# }
+# System.out.println(example);
+coder:
+ urn: "beam:coder:custom_window:v1"
+ components: [{urn: "beam:coder:interval_window:v1"}]
Review comment:
yep, using interval_window coder here makes it easier to verify in
standard coder test.
##########
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##########
@@ -939,6 +939,31 @@ message StandardCoders {
// Experimental.
STATE_BACKED_ITERABLE = 9 [(beam_urn) =
"beam:coder:state_backed_iterable:v1"];
+
+ // Encodes an arbitrary user defined window and its max timestamp
(inclusive).
+ // The encoding format is:
+ // maxTimestamp window
Review comment:
I think whether it should be length prefixed can be determined by runner
with coder overwrite, since we are not limiting the wrapped window coder type,
known coder can be also used(though it is unlikely in real use case, but it is
handy for unit tests) which don't need length prefix.
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectKinds.java
##########
@@ -21,6 +21,7 @@
class CloudObjectKinds {
static final String KIND_GLOBAL_WINDOW = "kind:global_window";
static final String KIND_INTERVAL_WINDOW = "kind:interval_window";
+ static final String KIND_CUSTOM_WINDOW = "kind:custom_window";
Review comment:
For translating it as a known coder on runner side.
##########
File path:
model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
##########
@@ -440,3 +441,25 @@ examples:
shardId: "",
key: "key"
}
+
+---
+
+# Java code snippet to generate example bytes:
+# TimestampPrefixingWindowCoder<IntervalWindow> coder =
TimestampPrefixingWindowCoder.of(IntervalWindowCoder.of());
+# Instant end = new Instant(-9223372036854410L);
+# Duration span = Duration.millis(365L);
+# IntervalWindow window = new IntervalWindow(end.minus(span), span);
+# byte[] bytes = CoderUtils.encodeToByteArray(coder, window);
+# String str = new String(bytes,
java.nio.charset.StandardCharsets.ISO_8859_1);
+# String example = "";
+# for(int i = 0; i < str.length(); i++){
+# example += CharUtils.unicodeEscaped(str.charAt(i));
+# }
+# System.out.println(example);
+coder:
+ urn: "beam:coder:custom_window:v1"
Review comment:
I think custom_window makes it a bit more clear that this is a coder for
window type. timestamp_prefixed also works as long as sdk and runner agree with
the urn I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
