[
https://issues.apache.org/jira/browse/BEAM-8876?focusedWorklogId=512339&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512339
]
ASF GitHub Bot logged work on BEAM-8876:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Nov/20 13:03
Start Date: 16/Nov/20 13:03
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #13287:
URL: https://github.com/apache/beam/pull/13287#discussion_r524046472
##########
File path: runners/spark/build.gradle
##########
@@ -41,10 +42,15 @@ evaluationDependsOn(":runners:core-java")
configurations {
validatesRunner
- hadoopVersion285
}
-def hadoop_285_version = '2.8.5'
+def hadoopVersions = [
Review comment:
Better :+1:
##########
File path: sdks/java/io/hadoop-common/build.gradle
##########
@@ -16,17 +16,21 @@
* limitations under the License.
*/
+import java.util.stream.Collectors
+
plugins { id 'org.apache.beam.module' }
applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop Common"
ext.summary = "Library to add shared Hadoop classes among Beam IOs."
-configurations {
- hadoopVersion285
-}
+def hadoopVersions = [
+ "285": "2.8.5",
+ "292": "2.9.2",
+ "2101": "2.10.1",
Review comment:
Are we API compatible to even include here "3.2.1"? Does it pass? if so
we can maybe include it in a subsequent PR (just thinking out loud for this and
the other Hadoop modules)
##########
File path: sdks/java/io/hadoop-common/build.gradle
##########
@@ -16,17 +16,21 @@
* limitations under the License.
*/
+import java.util.stream.Collectors
+
plugins { id 'org.apache.beam.module' }
applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop Common"
ext.summary = "Library to add shared Hadoop classes among Beam IOs."
-configurations {
- hadoopVersion285
-}
+def hadoopVersions = [
+ "285": "2.8.5",
+ "292": "2.9.2",
+ "2101": "2.10.1",
Review comment:
I also was thinking if we could define this globally so we don't have
this copy pasted everywhere as well as the hadoopVersions chunk, but my gradle
fu is not so high to purpose a way to do it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 512339)
Time Spent: 5.5h (was: 5h 20m)
> Tests to confirm Hadoop version compatibility
> ---------------------------------------------
>
> Key: BEAM-8876
> URL: https://issues.apache.org/jira/browse/BEAM-8876
> Project: Beam
> Issue Type: Test
> Components: build-system
> Reporter: Tomo Suzuki
> Assignee: Piotr Szuberski
> Priority: P3
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Add test(s) to confirm Hadoop version compatibility for hadoop_client library
> version against different users' environment.
> https://github.com/apache/beam/pull/10222#issuecomment-559158067
> [~iemejia] had an idea:
> {quote}After a quick review most of the major users and systems still support
> 2.7 and ‘even worse’ 2.6. So I was wondering how can we ensure that new
> patches on Hadoop modules don’t break backwards compatibility.
> One idea could be to add an extra set of tests that run the build for the
> provided previous versions of the dependencies. But I don’t have a clue how
> to do this in gradle. Do you by any chance? (Note that a fix for this will
> solve other compatibility issues we have with other systems and provided
> dependencies e.g. Spark and Kafka). {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
