Arun Mahadevan created SPARK-24697:
--------------------------------------
Summary: Fix the reported start and end offsets in streaming query
Key: SPARK-24697
URL: https://issues.apache.org/jira/browse/SPARK-24697
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.3.1
Reporter: Arun Mahadevan
Streaming query reports progress during each trigger (e.g. after runBatch in
MicrobatchExcecution). However the reported progress has wrong offsets since
the offsets are committed and committedOffsets is updated to the
availableOffsets before the progress is reported.
This leads to weird progress where startOffset and endOffsets are always the
same.
Sample output for Kafka source below. Here 11 rows are processed in the
microbatch however the start and end offsets are same.
{code:java}
{
"id" : "76bf5515-55be-46af-bc79-9fc92cc6d856",
"runId" : "b526f0f4-24bf-4ddc-b6e8-7b0cc83bdbe8",
...
"sources" : [ {
"description" : "KafkaV2[Subscribe[topic2]]",
"startOffset" : {
"topic2" : {
"0" : 44
}
},
"endOffset" : {
"topic2" : {
"0" : 44
}
},
"numInputRows" : 11,
"inputRowsPerSecond" : 1.099670098970309,
"processedRowsPerSecond" : 1.8829168093118795
} ],
...
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
