mosche commented on code in PR #27851:
URL: https://github.com/apache/beam/pull/27851#discussion_r1354717240
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java:
##########
@@ -485,8 +486,8 @@ public long getSplitBacklogBytes() {
* The checkpoint for an unbounded {@link CountingSource} is simply the last
value produced. The
* associated source object encapsulates the information needed to produce
the next value.
*/
- @DefaultCoder(AvroCoder.class)
- public static class CounterMark implements UnboundedSource.CheckpointMark {
+ @DefaultCoder(SerializableCoder.class)
Review Comment:
I would recommend creating a custom coder which is binary compatible to the
Avro based one. This is actually rather trivial, Avro does zigzag varint
encoding and the bytes of both longs in CounterMark are just written after
another without anything added.
Zigzag encoding is also trivial to implement, or using
`org.apache.beam.vendor.grpc.v1p54p0.com.google.protobuf.CodedOutputStream.encodeZigZag64`
(and Protobuf will always remain a core dependency):
```
public static long encodeZigZag64(final long n) {
return (n << 1) ^ (n >> 63);
}
public static long decodeZigZag64(final long n) {
return (n >>> 1) ^ -(n & 1);
}
```
And varint encoding / decoding is already implemented in
`org.apache.beam.sdk.util.VarInt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
