Jacob Ferriero created BEAM-9847:
------------------------------------
Summary: Verify If Triggering allows emitting eager results when
processing a single element in HL7v2IO.
Key: BEAM-9847
URL: https://issues.apache.org/jira/browse/BEAM-9847
Project: Beam
Issue Type: Task
Components: io-java-gcp
Reporter: Jacob Ferriero
Assignee: Jacob Ferriero
Due to the nature of the HL7v2 API, HL7v2IO.ListHL7v2Messages follows the
pattern of [paginating through all ListMessages results in a single
ProcessElement call.|#diff-9cd595f078378218ccc01ce7e19ca766R447]]
Upon testing with customer against HL7v2 store with 350k messages we observed
that the ListMessages transform was not outputting any elements "hanging" for a
long time (which was assumed to be the single thread paginating through all the
results).
We added the following triggering in hopes that it would emit early results:
{code:java}
.apply(
Window.<HL7v2Message>into(new GlobalWindows())
.triggering(
AfterWatermark.pastEndOfWindow()
.withEarlyFirings(
AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(Duration.standardSeconds(1))))
.discardingFiredPanes())
{code}
Our tests with this triggering seemed to indicate that it did not "hang" like
the first test and seemed to output a more steady stream of elements.
Reviewer states that bundles must be committed atomically so no output elements
of a (single process element call) can proceed to downstream stages until all
output elements for that process element call are ready.
There may be other things at play here. Will seek to reproduce in a way that
definitively confirms output elements can be eagerly output during the
execution of a single process element call before it completes.
CC: [~pabloem]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
