[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=353569&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353569 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 04/Dec/19 16:38 Start Date: 04/Dec/19 16:38 Worklog Time Spent: 10m Work Description: iemejia commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353569) Time Spent: 5h 40m (was: 5.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352632&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352632 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 13:56 Start Date: 03/Dec/19 13:56 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-561178703 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352632) Time Spent: 5.5h (was: 5h 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352627&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352627 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 13:54 Start Date: 03/Dec/19 13:54 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r353187496 ## File path: sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java ## @@ -45,6 +45,10 @@ private static final List HA_SCHEMES = Arrays.asList("hdfs", "webhdfs"); + // Using hard-coded value for compatibility, because hadoop-dfs-client:2.7's DFSConfigKeys holds + // this value while it is HdfsClientConfigKeys that holds the value in version 2.8. + private static final String CONFIG_KEY_DFS_NAMESERVICES = "dfs.nameservices"; Review comment: @iemejia Thanks. Would you merge this? I don't add the test in this PR. I created a followup issue: https://issues.apache.org/jira/browse/BEAM-8876 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352627) Time Spent: 5h 20m (was: 5h 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 5h 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352512&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352512 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 09:02 Start Date: 03/Dec/19 09:02 Worklog Time Spent: 10m Work Description: iemejia commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r353052981 ## File path: sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java ## @@ -45,6 +45,10 @@ private static final List HA_SCHEMES = Arrays.asList("hdfs", "webhdfs"); + // Using hard-coded value for compatibility, because hadoop-dfs-client:2.7's DFSConfigKeys holds + // this value while it is HdfsClientConfigKeys that holds the value in version 2.8. + private static final String CONFIG_KEY_DFS_NAMESERVICES = "dfs.nameservices"; Review comment: I think this fix makes this mergeable, do you plan to tackle the multi version testing as well? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352512) Time Spent: 5h 10m (was: 5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 5h 10m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352509&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352509 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 08:59 Start Date: 03/Dec/19 08:59 Worklog Time Spent: 10m Work Description: iemejia commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r353051346 ## File path: sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java ## @@ -45,6 +45,10 @@ private static final List HA_SCHEMES = Arrays.asList("hdfs", "webhdfs"); + // Using hard-coded value for compatibility, because hadoop-dfs-client:2.7's DFSConfigKeys holds + // this value while it is HdfsClientConfigKeys that holds the value in version 2.8. + private static final String CONFIG_KEY_DFS_NAMESERVICES = "dfs.nameservices"; Review comment: Yes you read my mind, I was going to subject that just as the best path forward. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352509) Time Spent: 5h (was: 4h 50m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352431 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 04:10 Start Date: 03/Dec/19 04:10 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r352975722 ## File path: sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java ## @@ -45,6 +45,10 @@ private static final List HA_SCHEMES = Arrays.asList("hdfs", "webhdfs"); + // Using hard-coded value for compatibility, because hadoop-dfs-client:2.7's DFSConfigKeys holds + // this value while it is HdfsClientConfigKeys that holds the value in version 2.8. + private static final String CONFIG_KEY_DFS_NAMESERVICES = "dfs.nameservices"; Review comment: Updated the source to use a string rather than a field of Hadoop class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352431) Time Spent: 4h 50m (was: 4h 40m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4h 50m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352430&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352430 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 03/Dec/19 04:03 Start Date: 03/Dec/19 04:03 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-560935075 > The hadoop modules (both hadoop-file-system and hadoop-format) are provided by the user (commonly by relying on the ones available in their clusters) Now I understand this problem better. The Hadoop client dependencies are marked as `provided`. With this PR, users who touch "hdfs://" scheme will have to upgrade their Hadoop client version (especially `org.apache.hadoop:hadoop-hdfs-client`) at least; otherwise they'll get ClassNotFoundError or NoClassDefFoundError for the new class `HdfsClientConfigKeys`. It also means that Beam's `HadoopFileSystemRegistrar` has not worked with `hadoop-hdfs-client` 2.8 or higher. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352430) Time Spent: 4h 40m (was: 4.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4h 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=352313&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352313 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 02/Dec/19 23:57 Start Date: 02/Dec/19 23:57 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-560935075 > The hadoop modules (both hadoop-file-system and hadoop-format) are provided by the user (commonly by relying on the ones available in their clusters) Now I understand this problem better. The Hadoop client dependencies are marked as `provided`. With this PR, users who touch "hdfs://" scheme will have to upgrade their Hadoop client version (especially `org.apache.hadoop:hadoop-hdfs-client`) at least; otherwise they'll get ClassNotFoundError or NoClassDefFoundError for the new class `HdfsClientConfigKeys`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352313) Time Spent: 4.5h (was: 4h 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=351970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351970 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 02/Dec/19 15:22 Start Date: 02/Dec/19 15:22 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-560442564 Thanks, everyone. Give me some time to confirm compatibility. (Now reading https://beam.apache.org/documentation/io/built-in/hadoop/ on how to use Hadoop from Beam) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 351970) Time Spent: 4h 20m (was: 4h 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4h 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=351054&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-351054 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 28/Nov/19 15:46 Start Date: 28/Nov/19 15:46 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559540157 Thanks Luke and the others for improving my explanation. One aspect however that differs in this case (compared with elasticsearch/flink) is that for both of them we 'include' the dependencies in the module. The hadoop modules (both hadoop-file-system and hadoop-format) are provided by the user (commonly by relying on the ones available in their clusters), so doing a test should be as easy as adding an extra test configuration in `hadoop-file-system` and `hadoop-format` with new parameters for the versions we want to test that we support. This is trivial to do in Maven `mvn test -Dhadoop.version=x.x.x` and I hope we could find a solution as simple as that one with gradle to not require the extra complexity of files and hierarchies that Elasticsearch and Flink have (but that’s a valid solution too). Sorry @suztomo don’t assume my comments/questions are in the sense that I don’t want to have this upgrade done (it is totally the opposite), or that I want to ‘feature creep’ this PR, but compatibility with what users run in their clusters is important and I am afraid of this being broken in the future as a consequence of this version change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 351054) Time Spent: 4h 10m (was: 4h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4h 10m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350942&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350942 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 28/Nov/19 12:22 Start Date: 28/Nov/19 12:22 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559474089 Yes - ping me and @mwalenia if you have any questions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350942) Time Spent: 4h (was: 3h 50m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 4h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350819 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 28/Nov/19 04:40 Start Date: 28/Nov/19 04:40 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559338734 Another example which is an IO is elasticsearch, where we run tests against multiple versions. Since there is a source incompatibility, you can split into modules (if necessary) in addition to running tests against multiple versions. - `beam-io-hadoop-format-lib` that works with all versions and just provides the main functionality And thin modules: - `beam-io-hadoop-format-2.6` - `beam-io-hadoop-format-2.7` - `beam-io-hadoop-format-2.8` where you can add references to new 2.8 features These might be able to be `build.gradle` only. And you may not actually need them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350819) Time Spent: 3h 40m (was: 3.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3h 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350820&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350820 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 28/Nov/19 04:40 Start Date: 28/Nov/19 04:40 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559338791 I believe @lgajowy is a good contact about these sorts of tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350820) Time Spent: 3h 50m (was: 3h 40m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3h 50m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350582 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 17:49 Start Date: 27/Nov/19 17:49 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559190042 Got it. Thank you for clarification. I'll check how Beam's Hadoop functionality is tested. Memo: Still I feel I need to verify whether Hadoop-client 2.8 is compatible with Hadoop cluster 2.7. - If it's compatible, then we can think about creating integration tests for multiple Hadoop versions (2.7 and 2.8). - If Hadoop-client 2.8 is not compatible with Haodop cluster 2.7, then some modules need to be split in the same manner as Flink's. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350582) Time Spent: 3.5h (was: 3h 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350575 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 17:33 Start Date: 27/Nov/19 17:33 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559184614 I believe what @iemejia is saying is that we should do something like [Flink](https://github.com/apache/beam/tree/master/runners/flink) where it runs tests using Flink 1.7, 1.8 and 1.9. In the Hadoop case we would run the tests using version 2.7 and 2.8 (and maybe 2.6) to show that our tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350575) Time Spent: 3h 20m (was: 3h 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3h 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350567&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350567 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 17:10 Start Date: 27/Nov/19 17:10 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559167049 The lifecycle of a product is to help to avoid supporting old product, and to guide decision like this. Still, we should be clear about compatibility. I don't use Hadoop and thus cannot test it. Let me send email to dev@ for anyone can verify the compatibility. (trying https://cloud.google.com/dataproc/ ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350567) Time Spent: 3h 10m (was: 3h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350555&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350555 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 16:47 Start Date: 27/Nov/19 16:47 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559167049 The lifecycle of a product is to help to avoid supporting old product, and to guide decision like this. Still, we should be clear about compatibility. I don't use Hadoop and thus cannot test it. Let me send email to dev@ for anyone can verify the compatibility. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350555) Time Spent: 3h (was: 2h 50m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 3h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350549&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350549 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 16:39 Start Date: 27/Nov/19 16:39 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559158067 Changes look good for the goal, however it seems that my comment about Hadoop 2.7.x being already EOL in upstream Hadoop was a bit optimistic. After a quick review most of the major users and systems still support 2.7 and ‘even worse’ 2.6. So I was wondering how can we ensure that new patches on Hadoop modules don’t break backwards compatibility. One idea could be to add an extra set of tests that run the build for the provided previous versions of the dependencies. But I don’t have a clue how to do this in gradle. Do you by any chance? (Note that a fix for this will solve other compatibility issues we have with other systems and provided dependencies e.g. Spark and Kafka). More details in [BEAM-4087](https://issues.apache.org/jira/browse/BEAM-4087) For the case of the current patch it is not backwards compatible at the source level (because it introduces a dep on a class that did not exist in previous hadoop versions) and the proposed tests may have helped to discover that (note that for this case it can be easily fixed by extracting the constant into a local String or just by keeping the warning on the old class). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350549) Time Spent: 2h 50m (was: 2h 40m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2h 50m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350526&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350526 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 16:24 Start Date: 27/Nov/19 16:24 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559158067 Changes look good for the goal, however it seems that my comment about Hadoop 2.7.x being already EOL in upstream Hadoop was a bit optimistic. After a quick review most of the major users and systems still support 2.7 and ‘even worse’ 2.6. So I was wondering how can we ensure that new patches on Hadoop modules don’t break backwards compatibility. One idea could be to add an extra set of tests that run the build for the provided previous versions of the dependencies. But I don’t have a clue how to do this in gradle. Do you by any chance? (Note that a fix for this will solve other compatibility issues we have with other systems and provided dependencies e.g. Spark and Kafka). More details in [BEAM-4087](https://issues.apache.org/jira/browse/BEAM-4087) For the case of the current patch it is not backwards compatible at the source level (because it introduces a dep on a class that did not exist in previous hadoop versions) and the proposed tests may have helped to discover that (but this can be easily fixed by extracting the constant into a local String or just by keeping the warning on the old class). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350526) Time Spent: 2h 40m (was: 2.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2h 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350450 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 14:54 Start Date: 27/Nov/19 14:54 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-559120337 The checks other than the problematic Python checks went successful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350450) Time Spent: 2.5h (was: 2h 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350196&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350196 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:05 Start Date: 27/Nov/19 04:05 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558919058 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350196) Time Spent: 2h 10m (was: 2h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350197 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:05 Start Date: 27/Nov/19 04:05 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558919145 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350197) Time Spent: 2h 20m (was: 2h 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350192&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350192 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:04 Start Date: 27/Nov/19 04:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558918937 Flink Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350192) Time Spent: 1.5h (was: 1h 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350195 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:04 Start Date: 27/Nov/19 04:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558919022 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350195) Time Spent: 2h (was: 1h 50m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 2h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350193&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350193 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:04 Start Date: 27/Nov/19 04:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558918951 Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350193) Time Spent: 1h 40m (was: 1.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350194&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350194 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 27/Nov/19 04:04 Start Date: 27/Nov/19 04:04 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558918999 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350194) Time Spent: 1h 50m (was: 1h 40m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350095&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350095 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 22:54 Start Date: 26/Nov/19 22:54 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r350897749 ## File path: sdks/java/io/hadoop-format/build.gradle ## @@ -73,6 +73,8 @@ dependencies { exclude group: "org.apache.spark", module: "spark-sql_2.10" exclude group: "org.apache.storm", module: "storm-core" } + // elasticsearch-hadoop 5.0.0 uses commons-httpclient's URIException + testCompile "commons-httpclient:commons-httpclient:3.1" Review comment: `Elasticsearch-hadoop` appear 12 lines above this line. Without commons-httpclient, HadoopFormatIOElasticTest fails with "NoClassDefFoundError: org/apache/commons/httpclient/URIException" ([detail](https://issues.apache.org/jira/browse/BEAM-8822?focusedCommentId=16981950&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16981950)) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350095) Time Spent: 1h 20m (was: 1h 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350094 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 22:52 Start Date: 26/Nov/19 22:52 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558851698 R: @iemejia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350094) Time Spent: 1h 10m (was: 1h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350058 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 21:39 Start Date: 26/Nov/19 21:39 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558828415 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350058) Time Spent: 50m (was: 40m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 50m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350060 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 21:39 Start Date: 26/Nov/19 21:39 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558828453 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350060) Time Spent: 1h (was: 50m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 1h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=350052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350052 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 21:34 Start Date: 26/Nov/19 21:34 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558826736 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 350052) Time Spent: 40m (was: 0.5h) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 40m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=349915&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349915 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 18:00 Start Date: 26/Nov/19 18:00 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#issuecomment-558748894 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 349915) Time Spent: 0.5h (was: 20m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=349914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349914 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 17:59 Start Date: 26/Nov/19 17:59 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222#discussion_r350897749 ## File path: sdks/java/io/hadoop-format/build.gradle ## @@ -73,6 +73,8 @@ dependencies { exclude group: "org.apache.spark", module: "spark-sql_2.10" exclude group: "org.apache.storm", module: "storm-core" } + // elasticsearch-hadoop 5.0.0 uses commons-httpclient's URIException + testCompile "commons-httpclient:commons-httpclient:3.1" Review comment: `Elasticsearch-hadoop` appear 12 lines above this line. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 349914) Time Spent: 20m (was: 10m) > Hadoop Client version 2.8 from 2.7 > -- > > Key: BEAM-8822 > URL: https://issues.apache.org/jira/browse/BEAM-8822 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Tomo Suzuki >Assignee: Tomo Suzuki >Priority: Major > Attachments: OGuVu0A18jJ.png > > Time Spent: 20m > Remaining Estimate: 0h > > [~iemejia] says: > bq. probably a quicker way forward is to unblock the bigtable issue is to > move our Hadoop dependency to Hadoop 2.8 given that Hadoop 2.7 is now EOL we > have a good reason to do so > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > The URL says > {quote}Following branches are EOL: > [2.0.x - 2.7.x]{quote} > https://issues.apache.org/jira/browse/BEAM-8569?focusedCommentId=16980532&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16980532 > About compatibility with other library: > Hadoop client 2.7 is not compatible with Guava > 21 because of > Objects.toStringHelper. Fortunately Hadoop client 2.8 removed the use of the > method > ([detail|https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/1028#issuecomment-557709027]). > 2.8.5 is the latest in 2.8.X. > !OGuVu0A18jJ.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8822) Hadoop Client version 2.8 from 2.7
[ https://issues.apache.org/jira/browse/BEAM-8822?focusedWorklogId=349913&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349913 ] ASF GitHub Bot logged work on BEAM-8822: Author: ASF GitHub Bot Created on: 26/Nov/19 17:59 Start Date: 26/Nov/19 17:59 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10222: [BEAM-8822] Hadoop client version 2.8.5 from 2.7 URL: https://github.com/apache/beam/pull/10222 Upgrading Hadoop client version 2.7 (EOL) to 2.8.5 (the latest in 2.8.X release). CC: @iemejia Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build Stat