[
https://issues.apache.org/jira/browse/BEAM-12164?focusedWorklogId=750748&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-750748
]
ASF GitHub Bot logged work on BEAM-12164:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 31/Mar/22 04:59
Start Date: 31/Mar/22 04:59
Worklog Time Spent: 10m
Work Description: hengfengli commented on a change in pull request #17200:
URL: https://github.com/apache/beam/pull/17200#discussion_r839168550
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java
##########
@@ -146,6 +156,17 @@ public TimestampRange initialRestriction(@Element
PartitionMetadata partition) {
return TimestampRange.of(startTimestamp, endTimestamp);
}
+ @GetSize
+ public double getSize(@Element PartitionMetadata partition, @Restriction
TimestampRange range)
+ throws Exception {
+ final BigDecimal timeGapInSeconds =
+ BigDecimal.valueOf(newTracker(partition,
range).getProgress().getWorkRemaining());
Review comment:
```
Do you think we could extract the progress tracking into its own class that
is used both here and in the restriction tracker? You would be able to create a
single instance in the setup method and re-use it?
```
I don't think we should do this. If you read the comment in
https://github.com/apache/beam/blob/bff4be488e12415cedc2e713f4e5129a18a14497/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L965-L967,
it also creates a new tracker and initializes it. It is actually the same.
Also, KafkaIO creates a new restriction tracker as well:
https://github.com/apache/beam/blob/8af69d7d4fe73fc94658f7d5b3d6721e969c7b1c/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java#L280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 750748)
Time Spent: 65h 20m (was: 65h 10m)
> SpannerIO Change Stream Connector
> ---------------------------------
>
> Key: BEAM-12164
> URL: https://issues.apache.org/jira/browse/BEAM-12164
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Thiago Nunes
> Assignee: Thiago Nunes
> Priority: P2
> Fix For: 2.37.0
>
> Time Spent: 65h 20m
> Remaining Estimate: 0h
>
> We would like to augment the existing Google Cloud SpannerIO connector
> ([https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java)]
> with the support for Spanner Change Streams (CDC). CDC support is just being
> implemented in Spanner and it will be exposed through a gRPC API. We will use
> such API to create a new SpannerIO.readChangeStream(...) implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
