[
https://issues.apache.org/jira/browse/BEAM-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Chauchot updated BEAM-12591:
------------------------------------
Description:
When spark 3 was introduced, *the spark structured streaming runner sources
were moved to spark/2 module*
([https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe])
and the build system was changed. Now spark runners are built per spark
version module (2 or 3) and the version specific sources are copied from
_spark-version/src_ to _spark-version/build/source-overrides_ and gradle source
sets are set to _main-spark src + spark-version/build/source-overrides_. The
problem is that _spark-version/src_ is outside of gradle source sets and thus
the IDE does not compile this source, that is not an issue when there is very
few classes but in the case of the spark structured streaming runner, *it is
the whole runner code that is rendered invisible to the IDE* which is
impracticable for runner development.
As ide project settings are gerenated with gradle import, changing the project
settings manually would not work as the settings would be erased with each
gradle import
was:
When spark 3 was introduced, *the spark structured streaming runner sources
were moved to spark/2 module*
(https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe)
and the build system was changed. Now spark runners are built per spark
version module (2 or 3) and the version specific sources are copied from
_spark-version/src_ to _spark-version/build/source-overrides_ and gradle source
sets are set to _main-spark src + spark-version/build/source-overrides_. The
problem is that _spark-version/src_ is outside of gradle source sets and thus
the IDE does not compile this source, that is not an issue when there is very
few classes but in the case of the spark structured streaming runner, *it is
the whole runner code that is rendered invisible to the IDE* which is
impracticable for runner development
We should just put back spark structured streaming sources to main spark
sources. This will not change the build as it is still done per spark-version
so deps will be ok.
> Spark Structured Streaming runner sources are not visible from the IDE
> ----------------------------------------------------------------------
>
> Key: BEAM-12591
> URL: https://issues.apache.org/jira/browse/BEAM-12591
> Project: Beam
> Issue Type: Bug
> Components: build-system, runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When spark 3 was introduced, *the spark structured streaming runner sources
> were moved to spark/2 module*
> ([https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe])
> and the build system was changed. Now spark runners are built per spark
> version module (2 or 3) and the version specific sources are copied from
> _spark-version/src_ to _spark-version/build/source-overrides_ and gradle
> source sets are set to _main-spark src +
> spark-version/build/source-overrides_. The problem is that
> _spark-version/src_ is outside of gradle source sets and thus the IDE does
> not compile this source, that is not an issue when there is very few classes
> but in the case of the spark structured streaming runner, *it is the whole
> runner code that is rendered invisible to the IDE* which is impracticable for
> runner development.
>
> As ide project settings are gerenated with gradle import, changing the
> project settings manually would not work as the settings would be erased with
> each gradle import
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
