Stas Levin created BEAM-2072:
--------------------------------
Summary: MicrobatchSource.reader stops reading after reaching
maxNumRecords for the first time
Key: BEAM-2072
URL: https://issues.apache.org/jira/browse/BEAM-2072
Project: Beam
Issue Type: Bug
Components: runner-spark
Affects Versions: Not applicable
Reporter: Stas Levin
Assignee: Stas Levin
{{MicrobatchSource.Reader}} stops reading further data after the first batch in
which it has reached {{maxNumRecords}}. Subsequent read request will check the
{{recordsRead}} and conclude that no data can be read.
The root cause being that {{recordsRead}} is maintained throughout the reader's
lifecycle, and can span multiple (spark) batches.
We should reset {{recordsRead}} every time the reader starts reading a fresh
batch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
