HeartSaVioR commented on a change in pull request #35513:
URL: https://github.com/apache/spark/pull/35513#discussion_r808404922
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -314,7 +314,27 @@ class MicroBatchExecution(
/* Initialize committed offsets to a committed batch, which at this
* is the second latest batch id in the offset log. */
if (latestBatchId != 0) {
+ /* SPARK-38033: In some unexpected cases, commit and offset are
inconsistent,
Review comment:
We wouldn't need this code comment. The log message we add should be
self-explanatory.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -314,7 +314,27 @@ class MicroBatchExecution(
/* Initialize committed offsets to a committed batch, which at this
* is the second latest batch id in the offset log. */
if (latestBatchId != 0) {
+ /* SPARK-38033: In some unexpected cases, commit and offset are
inconsistent,
+ * and offset is not written into HDFS continuously, as follows:
+ *
+ * commits
+ * /tmp/streaming_xxxxxxxx/commits/113256
+ * /tmp/streaming_xxxxxxxx/commits/113257
+ * offsets
+ * /tmp/streaming_xxxxxxxx/offsets/113257
+ * /tmp/streaming_xxxxxxxx/offsets/113259
+ *
+ * When we start the streaming program, batch ${latestBatchId - 1}
is 113258,
+ * but offsets 113258 doesn't exist, an exception will be
thrown,resulting in
+ * the program cannot be started. As an improvement, we could
probably do some
+ * simply analysis and give better error message. */
val secondLatestOffsets = offsetLog.get(latestBatchId - 1).getOrElse
{
+ logError(s"Please check the checkpoint, batch ${latestBatchId - 1}
doesn't exist. " +
Review comment:
I guess we can adjust the log message like below to give more context
and direct to what end users should do:
> s"The offset log for batch ${latestBatchId - 1} doesn't exist, which is
required to restart the query from the latest batch $latestBatchId from the
offset log. Please ensure there are two subsequent offset logs available for
the latest batch via manually deleting the offset file(s). Please also ensure
the latest batch for commit log is equal or one batch earlier than the latest
batch for offset log."
cc. @viirya @xuanyuanking Appreciate your refinement or better idea on
guidance message. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
