[
https://issues.apache.org/jira/browse/SPARK-36084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378934#comment-17378934
]
Hyukjin Kwon commented on SPARK-36084:
--------------------------------------
[~geekyouth] Spark 2.4 is EOL, and there will be no more releases in Spark 2.4.
Can you see if the issue still persists in other Spark versions of 3.x too?
> spark kafka offset missed some partition offset describle
> ---------------------------------------------------------
>
> Key: SPARK-36084
> URL: https://issues.apache.org/jira/browse/SPARK-36084
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: geekyouth
> Priority: Major
>
> Caused by: org.apache.spark.sql.streaming.StreamingQueryException:
> Set(rcf_adidas_wow_room_event-9, rcf_adidas_wow_room_event-8,
> rcf_adidas_wow_room_event-7, rcf_adidas_wow_room_event-6,
> rcf_adidas_wow_room_event-5) are gone. Some data may have been missed.
> Some data may have been lost because they are not available in Kafka any
> more; either the
> data was aged out by Kafka or the topic may have been deleted before all the
> data in the
> topic was processed. If you don't want your streaming query to fail on such
> cases, set the
> source option "failOnDataLoss" to "false".
> === Streaming Query ===
> Identifier: [id = b0e0f54e-e638-430e-9912-751234a6ce13, runId =
> 570e950b-edcb-40c4-99ea-fd4db958b225]
> Current Committed Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]:
> {"rcf_adidas_wow_room_event":
> {"8":3572,"2":3571,"5":3571,"4":3571,"7":3571,"1":3571,"9":3571,"3":3571,"6":3571,"0":3571}
> }}
> Current Available Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]:
> {"rcf_adidas_wow_room_event":
> {"2":3571,"4":3571,"1":3571,"3":3571,"0":3571}
> }}
>
> —
> My topic has 10 partitions , but the last offset dir records only 5
> partitions.
> After i delete the last error offset dir , the query runs success.
>
> But i do not understand what cause this error.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
