[
https://issues.apache.org/jira/browse/BEAM-14323?focusedWorklogId=759132&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759132
]
ASF GitHub Bot logged work on BEAM-14323:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Apr/22 11:38
Start Date: 20/Apr/22 11:38
Worklog Time Spent: 10m
Work Description: aromanenko-dev commented on code in PR #17389:
URL: https://github.com/apache/beam/pull/17389#discussion_r853989632
##########
runners/spark/spark_runner.gradle:
##########
@@ -49,67 +49,46 @@ configurations {
}
def hadoopVersions = [
- "285": "2.8.5",
Review Comment:
What caused so many indent-related fixes in this PR here and below?
##########
runners/spark/2/build.gradle:
##########
@@ -22,12 +22,7 @@ project.ext {
// Set the version of all Spark-related dependencies here.
spark_version = '2.4.8'
spark_scala_version = '2.11'
-
- // Version specific code overrides.
- main_source_overrides = ['./src/main/java']
- test_source_overrides = ['./src/test/java']
- main_resources_overrides = []
- test_resources_overrides = []
+ copySourceBase = true // enabled to use Spark 3 as primary dev version
Review Comment:
Is it a typo here? Should it be `Spark2`?
Issue Time Tracking
-------------------
Worklog Id: (was: 759132)
Time Spent: 50m (was: 40m)
> Improve IDE integration for Spark cross version builds
> ------------------------------------------------------
>
> Key: BEAM-14323
> URL: https://issues.apache.org/jira/browse/BEAM-14323
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Moritz Mack
> Assignee: Moritz Mack
> Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With the current build setup, developer experience is fairly poor when
> working with cross version builds for Spark (but also similarly for Flink):
> * Sources for version specific overrides are copied to a new location and
> references as gradle sources from there. First of all, this is totally
> unnecessary. These sources are not shared and should be used in place. But
> much more troublesome, the actual sources won't be resolved / checked by any
> IDE anymore and can't be properly worked on that way. Sadly for no reason at
> all ...
> * The actual shared resources on the other hand are referenced (added to
> srcDirs) in place. The IDE will randomly assign them to one Spark version
> module. Typically, for IntelliJ at least, that's the first (lower) one and
> not the one developers are actively working on.
> The suggested changes is:
> * Don't copy version specific overrides
> * Only copy shared sources conditionally based on a flag. This allows
> developers to disable copying to pick a primary version they intend to work
> on.
> Note: This is primary a cosmetic flag to improve IDE integration and has no
> impact on builds, even if all modules disable copying.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
