[jira] [Resolved] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved MAPREDUCE-5984. -- Resolution: Fixed native-task: reuse lz4 sources in hadoop-common --- Key: MAPREDUCE-5984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087308#comment-14087308 ] Binglin Chang commented on MAPREDUCE-5984: -- Thanks Sean and Todd, I have committed this. native-task: reuse lz4 sources in hadoop-common --- Key: MAPREDUCE-5984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087320#comment-14087320 ] Todd Lipcon commented on MAPREDUCE-5976: +1, looks good to me. I'll commit this momentarily native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087321#comment-14087321 ] Todd Lipcon commented on MAPREDUCE-5976: Actually it looks like this doesn't apply against tip of the branch. Is this patch based on another in-flight patch, Manu? native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6024) java.net.SocketTimeoutException in Fetcher caused jobs stuck for more than 1 hour
zhaoyunjiong created MAPREDUCE-6024: --- Summary: java.net.SocketTimeoutException in Fetcher caused jobs stuck for more than 1 hour Key: MAPREDUCE-6024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6024 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, task Reporter: zhaoyunjiong Assignee: zhaoyunjiong Priority: Critical 2014-08-04 21:09:42,356 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 2 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) 2014-08-04 21:09:42,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: fake.host.name:13562 freed by fetcher#33 in 180024ms 2014-08-04 21:09:55,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: Assigning fake.host.name:13562 with 3 to fetcher#33 2014-08-04 21:09:55,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 3 of 3 to fake.host.name:13562 to fetcher#33 2014-08-04 21:12:55,463 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 3 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) ... 2014-08-04 22:03:13,416 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: fake.host.name:13562 freed by fetcher#33 in 271081ms 2014-08-04 22:04:13,417 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: Assigning fake.host.name:13562 with 3 to fetcher#33 2014-08-04 22:04:13,417 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 3 of 3 to fake.host.name:13562 to fetcher#33 2014-08-04 22:07:13,449 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 3 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6024) java.net.SocketTimeoutException in Fetcher caused jobs stuck for more than 1 hour
[ https://issues.apache.org/jira/browse/MAPREDUCE-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated MAPREDUCE-6024: Attachment: MAPREDUCE-6024.patch This patch try to decrease the waiting time in three case: 1. default value for MAX_FETCH_FAILURES_NOTIFICATIONS is 3, if there is only one reduce, it will take very long time before fetchFailures reach the threshold. 2. decrease the default value MAX_ALLOWED_FETCH_FAILURES_FRACTION from 0.5 to 0.3, and make it configurable. 3. if fetcher failed to fetch data from a host 5 times(about 20 minutes), inform job tracker the failure. java.net.SocketTimeoutException in Fetcher caused jobs stuck for more than 1 hour - Key: MAPREDUCE-6024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6024 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, task Reporter: zhaoyunjiong Assignee: zhaoyunjiong Priority: Critical Attachments: MAPREDUCE-6024.patch 2014-08-04 21:09:42,356 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 2 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) 2014-08-04 21:09:42,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: fake.host.name:13562 freed by fetcher#33 in 180024ms 2014-08-04 21:09:55,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: Assigning fake.host.name:13562 with 3 to fetcher#33 2014-08-04 21:09:55,360 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 3 of 3 to fake.host.name:13562 to fetcher#33 2014-08-04 21:12:55,463 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 3 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) ... 2014-08-04 22:03:13,416 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: fake.host.name:13562 freed by fetcher#33 in 271081ms 2014-08-04 22:04:13,417 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: Assigning fake.host.name:13562 with 3 to fetcher#33 2014-08-04 22:04:13,417 INFO fetcher#33 org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 3 of 3 to fake.host.name:13562 to fetcher#33 2014-08-04 22:07:13,449 WARN fetcher#33 org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to fake.host.name:13562 with 3 map outputs java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:697) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:289) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated MAPREDUCE-5976: -- Attachment: mapreduce-5976-v4.txt native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976-v4.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087347#comment-14087347 ] Manu Zhang commented on MAPREDUCE-5976: --- sorry, my local branch was not updated. Recreated a patch against latest branch. native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976-v4.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087375#comment-14087375 ] Sean Zhong commented on MAPREDUCE-5976: --- Committed to branch at r1616115. Thanks! native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976-v4.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong resolved MAPREDUCE-5976. --- Resolution: Fixed Hadoop Flags: Reviewed commited to branch MR-2841 at r1616115 native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976-v4.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5978) native-task CompressTest failure on Ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong resolved MAPREDUCE-5978. --- Resolution: Fixed commited to branch MR-2841 at r1616116. Thanks, Manu. native-task CompressTest failure on Ubuntu -- Key: MAPREDUCE-5978 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5978 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Manu Zhang Attachments: mapreduce-5978.txt The MR-2841 branch fails the following unit tests on my box: CompressTest.testBzip2Compress:84 file compare result: if they are the same ,then return true expected:true but was:false CompressTest.testDefaultCompress:116 file compare result: if they are the same ,then return true expected:true but was:false We need to fix these before merging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087387#comment-14087387 ] Sean Zhong commented on MAPREDUCE-5987: --- Hi, I also cannot reproduce this either. We will try to find more different machines to test it. native-task: Unit test TestGlibCBug fails on ubuntu --- Key: MAPREDUCE-5987 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Sean Zhong Assignee: Sean Zhong Priority: Minor On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails [ RUN ] IFile.TestGlibCBug 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: Failure Value of: realKey Actual: 1127504685 Expected: expect[index] Which is: 4102672832 [ FAILED ] IFile.TestGlibCBug (0 ms) [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087530#comment-14087530 ] Hudson commented on MAPREDUCE-6014: --- FAILURE: Integrated in Hadoop-Yarn-trunk #635 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/635/]) MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616018) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestBlocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v1.patch Attach first version of the patch, changes: 1. bugfix use hadoop-mapreduce-client-common normal jar rather than test-jar 2. invoke nttest in maven native build 3. remove copy test.sh in pom.xml, use cmake configure_file instead, which can pass libjvm.so path to test script. Note that the run *mvn test -Pnative* still fail(or OOM) cause some java tests are still failing, I skip all java tests by running: *mvn test -Pnative -Dtest=ddd* Manually tested on macos and ubuntu, native tests run successfully. native-task: add native tests to maven and fix bug in pom.xml - Key: MAPREDUCE-6006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-6006.v1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087622#comment-14087622 ] Hudson commented on MAPREDUCE-6014: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1829 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1829/]) MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616018) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestBlocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087646#comment-14087646 ] Hudson commented on MAPREDUCE-6014: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1855 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1855/]) MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616018) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestBlocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088246#comment-14088246 ] Siqi Li commented on MAPREDUCE-4815: If anyone can give me some feedback, that would be great. FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v1.patch, MAPREDUCE-4815.v2.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6007) Create a new option for distcp -p which causes raw.* namespace extended attributes to not be preserved
[
https://issues.apache.org/jira/browse/MAPREDUCE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088300#comment-14088300
]
Andrew Wang commented on MAPREDUCE-6007:
bq. the only typo above is the last line which should be no raw xattrs are
preserved
If none of these flags are specified, AFAIK neither non-raw or raw xattrs are
preserved, i.e. no xattrs. Yes?
bq. I convinced myself that a relative path could never be relative to
/.reserved/raw since you can't set your working directory to that.
AFAIK you can set your wd to whatever you want, and you can have .. in
absolute paths too. We need to make sure that this path is fully normalized if
we're doing a prefix check. Paths from a FileStatus are normalized, but paths
coming from the user (like the ones coming out of a DistCpOptions) are suspect.
setTargetPathExists has one of these suspect checks.
Doc
* This is hard to read, could we expand this into a separate section and a new
table? I'd particularly like to see a fuller explanation of what happens with
different dst options.
CopyListing
* Let's improve the InvalidInputException message. Paths don't really specify
something, you could say starts with or something instead. We should also
print the target path.
* I don't quite understand this error either, why is a {{/.r/r}} src and
{{-pd}} not okay? The exception also mentions the target not starting with
{{/.r/r}}, but that's not part of the if check.
* Line longer than 80chars
* I expected to see a check that was if (-p || -px) !-pd src is /.r/r,
then also check that the dst supports xattrs and is /.r/r. I wish there was a
way to test that it's HDFS too, but looking for dest having /.r/r is probably
good enough.
CopyMapper
* Can we expand the block comment to say that toCopyListingFileStatus is used
to filter xattrs, and passing copyXAttrs in twice is okay because we already
did it earlier? The double passing looks weird, though logically correct.
DistCp:
* I really don't like setting the DISABLERAWXATTRS flag in setTargetPathExists,
since the expectation is that Options flags are set by the user. This method is
also not named such that doing this there makes sense. We have the target path
via the DistCpOptions, so let's be explicit and verbose with the checks
instead. This is quite possibly why the CopyListing check is confusing to me.
* To expand on the above, -px means preserving all xattrs, while -pxd means
preserving non-raw xattrs. Then we have {{toCopyListingFileStatus}} where the
{{preserveXAttrs}} parameter actually means preserve non-raw xattrs. This is
also definitely confusing...
DistCpOptionSwitch:
* XATTR is not a standard capitalization style, let's lower case it as xattr
here. XAttr isn't standard either, but that ship has sailed.
Test
* I'd like tests for weird src and dst paths, i.e. relative or containing ..s
* We could also test the no preserve flags behavior, that no xattrs at all
are preserved.
Create a new option for distcp -p which causes raw.* namespace extended
attributes to not be preserved
--
Key: MAPREDUCE-6007
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6007
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: distcp
Affects Versions: fs-encryption
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: MAPREDUCE-6007.001.patch, MAPREDUCE-6007.002.patch
As part of the Data at Rest Encryption work (HDFS-6134), we need to create a
new option for distcp which causes raw.* namespace extended attributes to not
be preserved. See the doc in HDFS-6509 for details. The default for this
option will be to preserve raw.* xattrs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088626#comment-14088626 ] Manu Zhang commented on MAPREDUCE-6006: --- Hi Binglin, do we need to revert changes to hadoop-maprecuce-client-common and hadoop-project brought in by MAPREDUCE-5991 ? Meanwhile, which java tests are still failing, could you give a list or open a jira for that ? native-task: add native tests to maven and fix bug in pom.xml - Key: MAPREDUCE-6006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-6006.v1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6025) native-task: fix native library distribution
Manu Zhang created MAPREDUCE-6025: - Summary: native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated MAPREDUCE-6025: -- Attachment: mapreduce-6025.txt native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088643#comment-14088643 ] Manu Zhang commented on MAPREDUCE-6025: --- [~tlipcon], [~decster] and [~clockfly], patch is attached. Please review. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5992) native-task test logs should not write to console
[ https://issues.apache.org/jira/browse/MAPREDUCE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088760#comment-14088760 ] Manu Zhang commented on MAPREDUCE-5992: --- Hi Todd, I've chatted a bit over this issue with Sean offline and we agreed that it is tricky to tie native log into log4j now. Could we mark this as won't fix ? native-task test logs should not write to console - Key: MAPREDUCE-5992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5992 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Most of our unit tests are configured with a log4j.properties test resource so they don't spout a bunch of output to the console. We need to do the same for native-task. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5993) native-task: simplify/remove dead code
[ https://issues.apache.org/jira/browse/MAPREDUCE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088767#comment-14088767 ] Manu Zhang commented on MAPREDUCE-5993: --- Hi Todd, as the above is fixed in MAPREDUCE-6000, any other parts which you suspect is dead code or could be simplified ? native-task: simplify/remove dead code -- Key: MAPREDUCE-5993 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5993 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon The native task code has a bunch of code in it which isn't related to the map output collector. I suspect much if this is dead code. Let's remove it before we merge, so that the amount of code we have to maintain going forward is more limited. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v2.patch Chages: Revert some changes done by MAPREDUCE-5991. To [~mauzhang]: generally the tests(LargeKVTest, KVTest) takes too much memory and time to run, in my default env it produces OOM, (those tests are important, but we may need to make it smaller when run as unit test), and the test output are written to project sources directory rather than target directory, which pollutes project structure. native-task: add native tests to maven and fix bug in pom.xml - Key: MAPREDUCE-6006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-6006.v1.patch, MAPREDUCE-6006.v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6006) native-task: add native tests to maven and fix bug in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6006: - Attachment: MAPREDUCE-6006.v3.patch update the patch, forget to pass skipTests env to nttest. native-task: add native tests to maven and fix bug in pom.xml - Key: MAPREDUCE-6006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6006 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-6006.v1.patch, MAPREDUCE-6006.v2.patch, MAPREDUCE-6006.v3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
