[
https://issues.apache.org/jira/browse/BEAM-9933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102734#comment-17102734
]
Ismaël Mejía edited comment on BEAM-9933 at 5/8/20, 4:54 PM:
-------------------------------------------------------------
Spark has a [checker of unsupported operations that matches
this|https://github.com/apache/spark/blob/0fb607ef3744e073a09423460a79e3325b24b6ad/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala#L128]
This also explained in the [Spark structured streaming programming
guide|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#unsupported-operations]
It can be somehow avoided by workaround it with bytecode manipulation but this
is not an ideal solution.
was (Author: iemejia):
Spark has a checker of unsupported operations that matches this
https://github.com/apache/spark/blob/0fb607ef3744e073a09423460a79e3325b24b6ad/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala#L128
It can be somehow avoided by workaround it with bytecode manipulation but this
is not an ideal solution.
> Support Streaming in Spark Structured Streaming Runner
> ------------------------------------------------------
>
> Key: BEAM-9933
> URL: https://issues.apache.org/jira/browse/BEAM-9933
> Project: Beam
> Issue Type: New Feature
> Components: runner-spark
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: structured-streaming
>
> The current version of the Spark Structured Streaming Runner only support
> Bounded IO. This ticket is to address the support of full Streaming.
> This issue is blocked because of issues on Spark with multiple aggregations
> SPARK-26655
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
