Moritz Mack created BEAM-14323:
----------------------------------
Summary: Improve IDE integration for Spark cross version builds
Key: BEAM-14323
URL: https://issues.apache.org/jira/browse/BEAM-14323
Project: Beam
Issue Type: Improvement
Components: runner-spark
Reporter: Moritz Mack
With the current build setup, developer experience is fairly poor when working
with cross version builds for Spark (but also similarly for Flink):
* Sources for version specific overrides are copied to a new location and
references as gradle sources from there. First of all, this is totally
unnecessary. These sources are not shared and should be used in place. But much
more troublesome, the actual sources won't be resolved / checked by any IDE
anymore and can't be properly worked on that way. Sadly for no reason at all ...
* The actual shared resources on the other hand are referenced (added to
srcDirs) in place. The IDE will randomly assign them to one Spark version
module. Typically, for IntelliJ at least, that's the first (lower) one and not
the one developers are actively working on.
The suggested changes is:
* Don't copy version specific overrides
* Only copy shared sources conditionally based on a flag. This allows
developers to disable copying to pick a primary version they intend to work on.
Note: This is primary a cosmetic flag to improve IDE integration and has no
impact on builds, even if all modules disable copying.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
