[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132669#comment-14132669 ] Hudson commented on MAPREDUCE-6025: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #679 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/679/]) MAPREDUCE-6025: distribute nativetask.so to right folder (Manu Zhang) (seanzhong: rev 6d39367f6a95383b634d79b6df35a278e0d89fe0) * hadoop-mapreduce-project/pom.xml * hadoop-dist/pom.xml native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132727#comment-14132727 ] Hudson commented on MAPREDUCE-6025: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1895 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1895/]) MAPREDUCE-6025: distribute nativetask.so to right folder (Manu Zhang) (seanzhong: rev 6d39367f6a95383b634d79b6df35a278e0d89fe0) * hadoop-mapreduce-project/pom.xml * hadoop-dist/pom.xml native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132766#comment-14132766 ] Hudson commented on MAPREDUCE-6025: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1870 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1870/]) MAPREDUCE-6025: distribute nativetask.so to right folder (Manu Zhang) (seanzhong: rev 6d39367f6a95383b634d79b6df35a278e0d89fe0) * hadoop-mapreduce-project/pom.xml * hadoop-dist/pom.xml native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098356#comment-14098356 ] Sean Zhong commented on MAPREDUCE-6025: --- Thanks, +1 native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098362#comment-14098362 ] Sean Zhong commented on MAPREDUCE-6025: --- Hi Binlingļ¼ Manu made patch 4 which change cp to cp -R, and other concerns. Are you ok with that? I will commit after you confirm. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098364#comment-14098364 ] Binglin Chang commented on MAPREDUCE-6025: -- Patch lgtm, +1 native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098367#comment-14098367 ] Sean Zhong commented on MAPREDUCE-6025: --- thanks, commited. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025-v4.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092379#comment-14092379 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, I see you patch, please add use *cp -R* rather than cp to preserve symlinks native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092394#comment-14092394 ] Sean Zhong commented on MAPREDUCE-6025: --- Can this config to moved to pom of sub project hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask? It seems not clean to have nativetask distributation configurations in top level pom.xml. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092417#comment-14092417 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Sean, as nativetask is part of mapreduce-client and mapreduce-client distribution need to include nativetask libraries, some config requires to be included in mapreduce-client pom(whether it is written explicitly like this patch or inherent from parent pom) native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090454#comment-14090454 ] Manu Zhang commented on MAPREDUCE-6025: --- Your approach is better. I've updated the patch accordingly which is modified from hadoop-project-dist script. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090526#comment-14090526 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, patch looks good, comments: 1. use space rather than tabs 2. I can see you reuse some code in hadoop-project-dist, I think *cp -R* is much simpler than *tar xxx*, look at hadoop-dist/pom.xml, which already uses cp -r native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091465#comment-14091465 ] Manu Zhang commented on MAPREDUCE-6025: --- thanks for the comment. A new patch is uploaded. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089027#comment-14089027 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, the patch only copies native libs but not jars. I am thinking another way of doing this, follow hadoop-assemblies rules, you can see the contents in hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089319#comment-14089319 ] Allen Wittenauer commented on MAPREDUCE-6025: - I was under the impression this was intentional. Some platforms can't build the native library code but can run the Java code. Therefore -Pdist required the -Pnative if you want to build and distribute the native code as well. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090144#comment-14090144 ] Manu Zhang commented on MAPREDUCE-6025: --- bq. the patch only copies native libs but not jars Hi Binglin, jars are copied over to hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/ as other sub projects under hadoop-mapreduce-client. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090260#comment-14090260
]
Binglin Chang commented on MAPREDUCE-6025:
--
bq. Hi Binglin, jars are copied over to
hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/ as other sub
projects under hadoop-mapreduce-client.
I see, hadoop-mapreduce-dist.xml do this automatically to all sub-modules,
still I think it's better to make native libs first copied in hadoop-mapreduce
project, then dist script well copy them to root project automatically, like
other projects(hadoop-common, hadoop-hdfs).
And the current patch will fail when build without native profile *mvn package
-Pdist -DskipTests* (cause native llib path does not exist)
{noformat}
[exec] $ cp -r
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/bin
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/etc
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/libexec
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/sbin
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0-SNAPSHOT/share
.
[exec] $ cp -r
/home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/target/usr/local/lib/lib*
./lib/native/
[exec]
[exec] Failed!
{noformat}
native-task: fix native library distribution
Key: MAPREDUCE-6025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: task
Reporter: Manu Zhang
Assignee: Manu Zhang
Attachments: mapreduce-6025.txt
currently running mvn install -Pdist fails and nativetask native library is
not distributed to hadoop tar
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090262#comment-14090262 ] Manu Zhang commented on MAPREDUCE-6025: --- yes, you are right. hadoop-common and hadoop-hdfs have inherited hadoop-project-dist which does the copy for them pre-dist. I'm thinking about adding a dist profile in nativetask which will do so. What do you think ? native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090267#comment-14090267 ] Binglin Chang commented on MAPREDUCE-6025: -- bq. adding a dist profile in nativetask which will do so Adding a dist profile in hadoop-mapreduce-project/pom.xml? native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090332#comment-14090332
]
Manu Zhang commented on MAPREDUCE-6025:
---
I mean adding a dist profile in hadoop-mapreduce-client-nativetask/pom.xml
which will create
target/hadoop-mapreduce-client-nativetask-${project.version}/lib/native
directory and copy native libs there (if native libs exist). Then the current
hadoop-dist script could do its job.
native-task: fix native library distribution
Key: MAPREDUCE-6025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: task
Reporter: Manu Zhang
Assignee: Manu Zhang
Attachments: mapreduce-6025.txt
currently running mvn install -Pdist fails and nativetask native library is
not distributed to hadoop tar
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[
https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090342#comment-14090342
]
Binglin Chang commented on MAPREDUCE-6025:
--
I am not sure this can work, the current hadoop-dist script only copy
sub-module jars, not native library paths, it does not have
target/hadoop-mapreduce-client-nativetask-${project.version} at all.
Add some script into hadoop-mapreduce-project/pom.xml dist profile like
hadoop-common/hadoop-hdfs(they inherent dist profile from
hadoop-project-dist/pom.xml) would work.
native-task: fix native library distribution
Key: MAPREDUCE-6025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: task
Reporter: Manu Zhang
Assignee: Manu Zhang
Attachments: mapreduce-6025.txt
currently running mvn install -Pdist fails and nativetask native library is
not distributed to hadoop tar
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088643#comment-14088643 ] Manu Zhang commented on MAPREDUCE-6025: --- [~tlipcon], [~decster] and [~clockfly], patch is attached. Please review. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
