[
https://issues.apache.org/jira/browse/BEAM-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-12591 started by Etienne Chauchot.
-----------------------------------------------
> Spark Structured Streaming runner sources should be put back to main src
> folder
> -------------------------------------------------------------------------------
>
> Key: BEAM-12591
> URL: https://issues.apache.org/jira/browse/BEAM-12591
> Project: Beam
> Issue Type: Bug
> Components: build-system, runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: P2
>
> When spark 3 was introduced, *the spark structured streaming runner sources
> were moved to spark/2 module*
> (https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe)
> and the build system was changed. Now spark runners are built per spark
> version module (2 or 3) and the version specific sources are copied from
> _spark-version/src_ to _spark-version/build/source-overrides_ and gradle
> source sets are set to _main-spark src +
> spark-version/build/source-overrides_. The problem is that
> _spark-version/src_ is outside of gradle source sets and thus the IDE does
> not compile this source, that is not an issue when there is very few classes
> but in the case of the spark structured streaming runner, *it is the whole
> runner code that is rendered invisible to the IDE* which is impracticable for
> runner development
>
> We should just put back spark structured streaming sources to main spark
> sources. This will not change the build as it is still done per spark-version
> so deps will be ok.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
