Jungtaek Lim created SPARK-51265:
------------------------------------
Summary: StackOverflowError or Internal Error have raised when
executing eagerlyExecuteCommands containing streaming source
Key: SPARK-51265
URL: https://issues.apache.org/jira/browse/SPARK-51265
Project: Spark
Issue Type: Bug
Components: SQL, Structured Streaming
Affects Versions: 4.0.0
Reporter: Jungtaek Lim
This seems to be a long standing issue.
We do not allow having streaming source in batch query, because the streaming
source has to be materialized to provide the actual data, and it has to be
executed with streaming engine.
This check is in QueryExecution.assertSupported, which is done between analyzed
and optimized. That said, if the query does not reach that path, it isn't going
to fail as expected.
In reality, I have seen either getting into StackOverflowError or
AnalysisException with "InternalError", which are not correct. This should be
marked as user error, with proper guidance of fixing the query.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
