[
https://issues.apache.org/jira/browse/FLINK-23327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377905#comment-17377905
]
Dawid Wysakowicz commented on FLINK-23327:
------------------------------------------
I guess, not the right project ;)
> Spark Structured Streaming runner sources should be put back to main src
> folder
> -------------------------------------------------------------------------------
>
> Key: FLINK-23327
> URL: https://issues.apache.org/jira/browse/FLINK-23327
> Project: Flink
> Issue Type: Bug
> Components: Build System
> Reporter: Etienne Chauchot
> Priority: Major
>
> When spark 3 was introduced, *the spark structured streaming runner sources
> were moved to spark/2 module*
> (https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe)
> and the build system was changed. Now spark runners are built per spark
> version module (2 or 3) and the version specific sources are copied from
> spark-_version/src_ to spark-_version/build/source-overrides_ and gradle
> source sets are set to _main spark src_ +
> spark-_version/build/source-overrides._ The problem is that
> spark-_version/src_ is outside of gradle source sets and thus *the IDE does
> not compile this source*, that is not an issue when there is very few classes
> but in the case of the spark structured streaming runner, it is the whole
> runner code that is rendered invisible to the IDE which is impracticable for
> runner development
>
> We should just put back spark structured streaming sources to main spark
> sources. This will not change the build as it is still done per spark-version
> so deps will be ok.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
