Etienne Chauchot created FLINK-23327:
----------------------------------------
Summary: Spark Structured Streaming runner sources should be put
back to main src folder
Key: FLINK-23327
URL: https://issues.apache.org/jira/browse/FLINK-23327
Project: Flink
Issue Type: Bug
Components: Build System
Reporter: Etienne Chauchot
When spark 3 was introduced, *the spark structured streaming runner sources
were moved to spark/2 module*
(https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe)
and the build system was changed. Now spark runners are built per spark
version module (2 or 3) and the version specific sources are copied from
spark-_version/src_ to spark-_version/build/source-overrides_ and gradle source
sets are set to _main spark src_ + spark-_version/build/source-overrides._ The
problem is that spark-_version/src_ is outside of gradle source sets and thus
*the IDE does not compile this source*, that is not an issue when there is very
few classes but in the case of the spark structured streaming runner, it is the
whole runner code that is rendered invisible to the IDE which is impracticable
for runner development
We should just put back spark structured streaming sources to main spark
sources. This will not change the build as it is still done per spark-version
so deps will be ok.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
