[jira] [Commented] (FLINK-13945) Vendor-repos Maven profile doesn't exist in flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924272#comment-16924272
]
Till Rohrmann commented on FLINK-13945:
---
I prefer the latter approach as well.
> Vendor-repos Maven profile doesn't exist in flink-shaded
>
>
> Key: FLINK-13945
> URL: https://issues.apache.org/jira/browse/FLINK-13945
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Affects Versions: shaded-7.0, shaded-8.0, shaded-9.0
>Reporter: Elise Ramé
>Priority: Major
>
> According to
> [documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#custom--vendor-specific-versions],
> to build Flink against a vendor specific Hadoop version it is necessary to
> build flink-shaded against this version first :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=
> {code}
> vendor-repos profile has to be activated to include Hadoop vendors
> repositories.
> But Maven cannot find expected Hadoop dependencies and returns an error
> because vendor-repos profile isn't defined in flink-shaded.
> Example using flink-shaded 8.0 and HDP 2.6.5 Hadoop version :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.7.3.2.6.5.0-292
> {code}
> {code:bash}
> [INFO] ---< org.apache.flink:flink-shaded-hadoop-2
> >---
> [INFO] Building flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0
> [10/11]
> [INFO] [ jar
> ]-
> [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292
> is missing, no dependency information available
> [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] flink-shaded 8.0 ... SUCCESS [ 2.122
> s]
> [INFO] flink-shaded-force-shading 8.0 . SUCCESS [ 0.607
> s]
> [INFO] flink-shaded-asm-7 7.1-8.0 . SUCCESS [ 0.667
> s]
> [INFO] flink-shaded-guava-18 18.0-8.0 . SUCCESS [ 1.452
> s]
> [INFO] flink-shaded-netty-4 4.1.39.Final-8.0 .. SUCCESS [ 4.597
> s]
> [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-8.0 SUCCESS [ 0.620
> s]
> [INFO] flink-shaded-jackson-parent 2.9.8-8.0 .. SUCCESS [ 0.018
> s]
> [INFO] flink-shaded-jackson-2 2.9.8-8.0 ... SUCCESS [ 0.914
> s]
> [INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-8.0 . SUCCESS [ 0.627
> s]
> [INFO] flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0 FAILURE [ 0.047
> s]
> [INFO] flink-shaded-hadoop-2-uber 2.7.3.2.6.5.0-292-8.0 ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 11.947 s
> [INFO] Finished at: 2019-09-03T16:52:59+02:00
> [INFO]
>
> [WARNING] The requested profile "vendor-repos" could not be activated because
> it does not exist.
> [ERROR] Failed to execute goal on project flink-shaded-hadoop-2: Could not
> resolve dependencies for project
> org.apache.flink:flink-shaded-hadoop-2:jar:2.7.3.2.6.5.0-292-8.0: The
> following artifacts could not be resolved:
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292: Failure to find
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and
[jira] [Commented] (FLINK-13945) Vendor-repos Maven profile doesn't exist in flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924259#comment-16924259
]
Chesnay Schepler commented on FLINK-13945:
--
It depends. We can either add the vendor profiles to flink-shaded, or instruct
users to setup these repositories in their local environments.
I'd prefer the latter; it's vastly more powerful and less work for us.
> Vendor-repos Maven profile doesn't exist in flink-shaded
>
>
> Key: FLINK-13945
> URL: https://issues.apache.org/jira/browse/FLINK-13945
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Affects Versions: shaded-7.0, shaded-8.0, shaded-9.0
>Reporter: Elise Ramé
>Priority: Major
>
> According to
> [documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#custom--vendor-specific-versions],
> to build Flink against a vendor specific Hadoop version it is necessary to
> build flink-shaded against this version first :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=
> {code}
> vendor-repos profile has to be activated to include Hadoop vendors
> repositories.
> But Maven cannot find expected Hadoop dependencies and returns an error
> because vendor-repos profile isn't defined in flink-shaded.
> Example using flink-shaded 8.0 and HDP 2.6.5 Hadoop version :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.7.3.2.6.5.0-292
> {code}
> {code:bash}
> [INFO] ---< org.apache.flink:flink-shaded-hadoop-2
> >---
> [INFO] Building flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0
> [10/11]
> [INFO] [ jar
> ]-
> [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292
> is missing, no dependency information available
> [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] flink-shaded 8.0 ... SUCCESS [ 2.122
> s]
> [INFO] flink-shaded-force-shading 8.0 . SUCCESS [ 0.607
> s]
> [INFO] flink-shaded-asm-7 7.1-8.0 . SUCCESS [ 0.667
> s]
> [INFO] flink-shaded-guava-18 18.0-8.0 . SUCCESS [ 1.452
> s]
> [INFO] flink-shaded-netty-4 4.1.39.Final-8.0 .. SUCCESS [ 4.597
> s]
> [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-8.0 SUCCESS [ 0.620
> s]
> [INFO] flink-shaded-jackson-parent 2.9.8-8.0 .. SUCCESS [ 0.018
> s]
> [INFO] flink-shaded-jackson-2 2.9.8-8.0 ... SUCCESS [ 0.914
> s]
> [INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-8.0 . SUCCESS [ 0.627
> s]
> [INFO] flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0 FAILURE [ 0.047
> s]
> [INFO] flink-shaded-hadoop-2-uber 2.7.3.2.6.5.0-292-8.0 ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 11.947 s
> [INFO] Finished at: 2019-09-03T16:52:59+02:00
> [INFO]
>
> [WARNING] The requested profile "vendor-repos" could not be activated because
> it does not exist.
> [ERROR] Failed to execute goal on project flink-shaded-hadoop-2: Could not
> resolve dependencies for project
> org.apache.flink:flink-shaded-hadoop-2:jar:2.7.3.2.6.5.0-292-8.0: The
> following artifacts could not be resolved:
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292: Failure to find
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of
[jira] [Commented] (FLINK-13945) Vendor-repos Maven profile doesn't exist in flink-shaded
[
https://issues.apache.org/jira/browse/FLINK-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922285#comment-16922285
]
Till Rohrmann commented on FLINK-13945:
---
Thanks for reporting this issue [~Elise Ramé].
[~Zentol] I guess it could be helpful to update the documentation how to build
flink shaded against a vendor specific Hadoop version and update the Flink
documentation accordingly.
> Vendor-repos Maven profile doesn't exist in flink-shaded
>
>
> Key: FLINK-13945
> URL: https://issues.apache.org/jira/browse/FLINK-13945
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Affects Versions: shaded-7.0, shaded-8.0, shaded-9.0
>Reporter: Elise Ramé
>Priority: Major
>
> According to
> [documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#custom--vendor-specific-versions],
> to build Flink against a vendor specific Hadoop version it is necessary to
> build flink-shaded against this version first :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=
> {code}
> vendor-repos profile has to be activated to include Hadoop vendors
> repositories.
> But Maven cannot find expected Hadoop dependencies and returns an error
> because vendor-repos profile isn't defined in flink-shaded.
> Example using flink-shaded 8.0 and HDP 2.6.5 Hadoop version :
> {code:bash}
> mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.7.3.2.6.5.0-292
> {code}
> {code:bash}
> [INFO] ---< org.apache.flink:flink-shaded-hadoop-2
> >---
> [INFO] Building flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0
> [10/11]
> [INFO] [ jar
> ]-
> [WARNING] The POM for org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292
> is missing, no dependency information available
> [WARNING] The POM for org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292 is
> missing, no dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [WARNING] The POM for
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292 is missing, no
> dependency information available
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] flink-shaded 8.0 ... SUCCESS [ 2.122
> s]
> [INFO] flink-shaded-force-shading 8.0 . SUCCESS [ 0.607
> s]
> [INFO] flink-shaded-asm-7 7.1-8.0 . SUCCESS [ 0.667
> s]
> [INFO] flink-shaded-guava-18 18.0-8.0 . SUCCESS [ 1.452
> s]
> [INFO] flink-shaded-netty-4 4.1.39.Final-8.0 .. SUCCESS [ 4.597
> s]
> [INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-8.0 SUCCESS [ 0.620
> s]
> [INFO] flink-shaded-jackson-parent 2.9.8-8.0 .. SUCCESS [ 0.018
> s]
> [INFO] flink-shaded-jackson-2 2.9.8-8.0 ... SUCCESS [ 0.914
> s]
> [INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-8.0 . SUCCESS [ 0.627
> s]
> [INFO] flink-shaded-hadoop-2 2.7.3.2.6.5.0-292-8.0 FAILURE [ 0.047
> s]
> [INFO] flink-shaded-hadoop-2-uber 2.7.3.2.6.5.0-292-8.0 ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 11.947 s
> [INFO] Finished at: 2019-09-03T16:52:59+02:00
> [INFO]
>
> [WARNING] The requested profile "vendor-repos" could not be activated because
> it does not exist.
> [ERROR] Failed to execute goal on project flink-shaded-hadoop-2: Could not
> resolve dependencies for project
> org.apache.flink:flink-shaded-hadoop-2:jar:2.7.3.2.6.5.0-292-8.0: The
> following artifacts could not be resolved:
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-client:jar:2.7.3.2.6.5.0-292,
> org.apache.hadoop:hadoop-yarn-common:jar:2.7.3.2.6.5.0-292: Failure to find
> org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.5.0-292 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack
