[jira] [Commented] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043748#comment-16043748 ] Apache Spark commented on SPARK-13933: -- User 'wangyum' has created a pull request for this issue: https://github.com/apache/spark/pull/18247 > hadoop-2.7 profile's curator version should be 2.7.1 > > > Key: SPARK-13933 > URL: https://issues.apache.org/jira/browse/SPARK-13933 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 2.0.0 >Reporter: Steve Loughran >Priority: Minor > > This is pretty minor, more due diligence than any binary compatibility. > # the {{hadoop-2.7}} profile declares the curator version to be 2.6.0 > # the actual hadoop-2.7.1 dependency is of curator 2.7.1; this came from > HADOOP-11492 > For consistency, the profile can/should be changed. However, note that as > well as some incompatibilities defined in HADOOP-11492; the version of Guava > that curator asserts a need for is 15.x. HADOOP-11612 showed what needed to > be done to address compatibility problems there; one of the Curator classes > had to be forked to make compatible with guava 11+ -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197217#comment-15197217 ] Sean Owen commented on SPARK-13933: --- Tachyon isn't used in the project anymore in 2.x anyway, so that much is safe for 2.x. It's probably fine. > hadoop-2.7 profile's curator version should be 2.7.1 > > > Key: SPARK-13933 > URL: https://issues.apache.org/jira/browse/SPARK-13933 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 2.0.0 >Reporter: Steve Loughran >Priority: Minor > > This is pretty minor, more due diligence than any binary compatibility. > # the {{hadoop-2.7}} profile declares the curator version to be 2.6.0 > # the actual hadoop-2.7.1 dependency is of curator 2.7.1; this came from > HADOOP-11492 > For consistency, the profile can/should be changed. However, note that as > well as some incompatibilities defined in HADOOP-11492; the version of Guava > that curator asserts a need for is 15.x. HADOOP-11612 showed what needed to > be done to address compatibility problems there; one of the Curator classes > had to be forked to make compatible with guava 11+ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197206#comment-15197206 ] Steve Loughran commented on SPARK-13933: No custom guava built. Hadoop dealt with the problem by -experimentally building curator with guava.version set to 11 to see what broke (me: HADOOP-11102 ) -fixing the one needed operation by duplicating the code in hadoop code with it fixed to work with guava 11. There's no problem AFAIK; the reason for the hadoop update was to get some ZK fixes related to NN HA in. Even so, having a library that is known to be incompatible with a shared dependency is dangerous; I wasn't happy with the change going in without much discussion, and if HADOOP-11102 hadn't reassured me about compatibility, I'd have reverted things. The Hadoop change notes flag the API changes as: * Binary incompatible change: {{org.apache.curator.utils.PathUtils.validatePath(String)}} changed return types. Downstream users of this method will need to recompile. *Not directly used in spark codebase* * Source incompatible change: {{org.apache.curator.framework.recipes.shared.SharedCountReader}} added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile. *Not directly used in spark codebase* * Source incompatible change: {{org.apache.curator.framework.recipes.shared.SharedValueReader}} added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile. *Not directly used in spark codebase* As none of them are directly used, there's no direct risk of things breaking if spark depended on 2.6 and someone ran it with 2.7.1. Tachyon-client does use curator; someone would need to see its versioning/compatibility there; it may already be on a later version. How about I just add a one-liner to the profile saying "deliberately held back for guava reasons" > hadoop-2.7 profile's curator version should be 2.7.1 > > > Key: SPARK-13933 > URL: https://issues.apache.org/jira/browse/SPARK-13933 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 2.0.0 >Reporter: Steve Loughran >Priority: Minor > > This is pretty minor, more due diligence than any binary compatibility. > # the {{hadoop-2.7}} profile declares the curator version to be 2.6.0 > # the actual hadoop-2.7.1 dependency is of curator 2.7.1; this came from > HADOOP-11492 > For consistency, the profile can/should be changed. However, note that as > well as some incompatibilities defined in HADOOP-11492; the version of Guava > that curator asserts a need for is 15.x. HADOOP-11612 showed what needed to > be done to address compatibility problems there; one of the Curator classes > had to be forked to make compatible with guava 11+ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13933) hadoop-2.7 profile's curator version should be 2.7.1
[ https://issues.apache.org/jira/browse/SPARK-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197163#comment-15197163 ] Sean Owen commented on SPARK-13933: --- The curator update itself seems fine but the Guava thing is unfortunate. So we'd have to depend on a custom Guava build? Is there an actual problem to not updating the curator dep -- if they're API compatible it shouldn't matter really. > hadoop-2.7 profile's curator version should be 2.7.1 > > > Key: SPARK-13933 > URL: https://issues.apache.org/jira/browse/SPARK-13933 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 2.0.0 >Reporter: Steve Loughran >Priority: Minor > > This is pretty minor, more due diligence than any binary compatibility. > # the {{hadoop-2.7}} profile declares the curator version to be 2.6.0 > # the actual hadoop-2.7.1 dependency is of curator 2.7.1; this came from > HADOOP-11492 > For consistency, the profile can/should be changed. However, note that as > well as some incompatibilities defined in HADOOP-11492; the version of Guava > that curator asserts a need for is 15.x. HADOOP-11612 showed what needed to > be done to address compatibility problems there; one of the Curator classes > had to be forked to make compatible with guava 11+ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org