[
https://issues.apache.org/jira/browse/BEAM-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Chauchot updated BEAM-12591:
------------------------------------
Summary: Spark Structured Streaming runner: code source is not visible from
the IDE (was: Spark Structured Streaming runner: sources are not visible from
the IDE)
> Spark Structured Streaming runner: code source is not visible from the IDE
> --------------------------------------------------------------------------
>
> Key: BEAM-12591
> URL: https://issues.apache.org/jira/browse/BEAM-12591
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: P2
> Labels: structured-streaming
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When spark 3 was introduced, *the spark structured streaming runner sources
> were moved to spark/2 module*
> ([https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe])
> and the build system was changed. Now spark runners are built per spark
> version module (2 or 3) and the version specific sources are copied from
> _spark-version/src_ to _spark-version/build/source-overrides_ and gradle
> source sets are set to _main-spark src +
> spark-version/build/source-overrides_. The problem is that
> _spark-version/src_ is outside of gradle source sets and thus the IDE does
> not compile this source, that is not an issue when there is very few classes
> but in the case of the spark structured streaming runner, *it is the whole
> runner code that is rendered invisible to the IDE* which is impracticable for
> runner development.
>
> As ide project settings are gerenated with gradle import, changing the
> project settings manually would not work as the settings would be erased with
> each gradle import
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
