[
https://issues.apache.org/jira/browse/SPARK-39412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552172#comment-17552172
]
Apache Spark commented on SPARK-39412:
--------------------------------------
User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/36824
> IllegalStateException from connector does not work well with error class
> framework
> ----------------------------------------------------------------------------------
>
> Key: SPARK-39412
> URL: https://issues.apache.org/jira/browse/SPARK-39412
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.3.0
> Reporter: Jungtaek Lim
> Assignee: Max Gekk
> Priority: Blocker
> Fix For: 3.4.0, 3.3.1
>
> Attachments: kafka-dataloss-error-msg-in-spark-3-2.log,
> kafka-dataloss-error-msg-in-spark-3-3-or-master.log
>
>
> With SPARK-39346, Spark SQL binds several exceptions to the internal error,
> and produces different guidance on dealing with the exception. This assumes
> these exceptions are only used for noticing internal bugs.
> This applies to "connectors" as well, and introduces side-effect on the error
> log. For Kafka data source, it is a breaking and unacceptable change, because
> there is an important use case Kafka data source determines a case of
> "dataloss", and throws IllegalStateException with instruction message on
> workaround.
> I mentioned this as "important" use case, because it can even happen with
> some valid scenarios - streaming query has some maintenance period and
> Kafka's retention on topic removes some records in the meanwhile.
> Two problems arise:
> 1) This does not mean Spark has a bug and end users have to report, hence the
> guidance message on internal error is misleading.
> 2) Most importantly, instruction message is shown after a long stack trace.
> With the modification of existing test suite, I see the message being
> appeared in "line 90" of the error log.
> We should roll the right error message back, at least for Kafka's case.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
