jaketf commented on a change in pull request #11596:
URL: https://github.com/apache/beam/pull/11596#discussion_r421847633
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
##########
@@ -472,24 +548,120 @@ public void initClient() throws IOException {
this.client = new HttpHealthcareApiClient();
}
+ @GetInitialRestriction
+ public OrderedTimeRange getEarliestToLatestRestriction(@Element String
hl7v2Store)
+ throws IOException {
+ from = this.client.getEarliestHL7v2SendTime(hl7v2Store, this.filter);
+ // filters are [from, to) to match logic of OffsetRangeTracker but need
latest element to be
+ // included in results set to add an extra ms to the upper bound.
+ to = this.client.getLatestHL7v2SendTime(hl7v2Store, this.filter).plus(1);
+ return new OrderedTimeRange(from, to);
+ }
+
+ @NewTracker
+ public OrderedTimeRangeTracker newTracker(@Restriction OrderedTimeRange
timeRange) {
+ return timeRange.newTracker();
+ }
+
+ @SplitRestriction
+ public void split(
+ @Restriction OrderedTimeRange timeRange,
OutputReceiver<OrderedTimeRange> out) {
+ // TODO(jaketf) How to pick optimal values for desiredNumOffsetsPerSplit
?
Review comment:
My intuition says hourly splits seems like a more reasonable starting
point than day but I'm unsure about this without testing on a few datasets.
However for super large time ranges 10 years = 87,600 splits. naturally this
explodes if we think about even smaller time ranges.
For example, I assume the initial collection of splits returned in
SplitRestriction has to fit in memory?
Or even just the loop to define all these might take a significant amount of
time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
