[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Attachment: MAPREDUCE-6525.001.patch I attached the 001. The copy and logging of configuration should be done after all services are started. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Description: MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with configuration of existing MiniMRYarnCluster but the address of HistoryServer is properly set. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Status: Patch Available (was: Open) > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976029#comment-14976029
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
{{MiniMRYarnClusterAdapter#restart}} creates new MiniMRYarnCluster with
configuration of existing MiniMRYarnCluster. MiniMRYarnCluster tries to copy
the value of {{JHAdminConfig.MR_HISTORY_ADDRESS}} in the configuration of
JobHistoryServer (updated by using {{Configuration#updateConnectAddr}}) in
startup.
{code}
//need to do this because historyServer.init creates a new Configuration
getConfig().set(JHAdminConfig.MR_HISTORY_ADDRESS,
historyServer.getConfig().get(JHAdminConfig.MR_HISTORY_ADDRESS));
MRWebAppUtil.setJHSWebappURLWithoutScheme(getConfig(),
MRWebAppUtil.getJHSWebappURLWithoutScheme(historyServer.getConfig()));
{code}
According to the log of minicluster, the updated value seems not yet to be
visible at above point.
before restart:
{noformat}
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(248)) - MiniMRYARN ResourceManager
address: centos7:0
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(250)) - MiniMRYARN ResourceManager web
address: centos7:0
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(252)) - MiniMRYARN HistoryServer address:
centos7:0
2015-10-27 17:27:52,826 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(254)) - MiniMRYARN HistoryServer web
address: localhost:0
{noformat}
after restart:
{noformat}
2015-10-27 17:27:55,330 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(198)) - MiniMRYARN ResourceManager
address: localhost:45359
2015-10-27 17:27:55,330 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(200)) - MiniMRYARN ResourceManager web
address: centos7:34289
2015-10-27 17:27:55,331 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(202)) - MiniMRYARN HistoryServer address:
localhost:52822
2015-10-27 17:27:55,331 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(204)) - MiniMRYARN HistoryServer web
address: localhost:57028
{noformat}
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved MAPREDUCE-6524. - Resolution: Duplicate The cause seems to be same with MAPREDUCE-6525 and the patch of it fixed this too. I'm closing this as duplicate. > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > > The historyClient tried to connect to port 0 and failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976091#comment-14976091
]
Masatake Iwasaki commented on MAPREDUCE-6521:
-
The failure of TestMRJobsWithHistoryService and TestMiniMRClientCluster is not
related to the patch. I filed MAPREDUCE-6524 and MAPREDUCE-6525 for them.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6524: Description: The historyClient tried to connect to port 0 and failed. > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > > The historyClient tried to connect to port 0 and failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[
https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976083#comment-14976083
]
Masatake Iwasaki commented on MAPREDUCE-6524:
-
{noformat}
Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 95.533 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
testJobHistoryData(org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService)
Time elapsed: 94.426 sec <<< ERROR!
java.io.IOException: java.net.ConnectException: Call From centos7/127.0.0.1 to
centos7:0 failed on connection exception: java.net.ConnectException: Connection
refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
...
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.testJobHistoryData(TestMRJobsWithHistoryService.java:153)
{noformat}
> Fix intermittent test failure of
> TestMRJobsWithHistoryService.testJobHistoryData
>
>
> Key: MAPREDUCE-6524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> The historyClient tried to connect to port 0 and failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976320#comment-14976320
]
Hadoop QA commented on MAPREDUCE-6525:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 6m 22s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 34s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 122m 38s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 140m 57s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestNonExistentJob |
| Timed out tests | org.apache.hadoop.mapred.TestJobName |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768937/MAPREDUCE-6525.001.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 96677be |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6090/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6090/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6090/console |
This message was automatically generated.
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: MAPREDUCE-6525.001.patch
>
>
> MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with
> configuration of existing MiniMRYarnCluster but the address of HistoryServer
> is properly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976287#comment-14976287
]
Akira AJISAKA commented on MAPREDUCE-6499:
--
Thanks [~linyiqun] for creating the patch. Tried your patch and got the
following error:
{noformat}
DataTables warning (table id = 'jobs'): Requested unknown parameter '21' from
the data source for row 0
{noformat}
Probably this is because elapsed time is missing between reduces completed and
uber job. Would you fix it?
{code:title=HsJobsBlock.java}
+ .append(String.valueOf(job.getReducesCompleted())).append("\",\"")
+ .append(job.isUber()).append("\",\"")
{code}
> Add More Informations Of Jobs In JobHistory Main Page
> -
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Trivial
> Attachments: HADOOP-MAPREDUCE-6499.2.patch,
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch,
> HADOOP-MAPREDUCE-6499.patch
>
>
> Now in JobHistory Main Page show too little information about finished
> jobs,even don't have the job's running total time,only
> startTime,finishedTime.So,in jobHistory main page,we can add more
> informations about jobs, that we can better analyze jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976679#comment-14976679 ] Masatake Iwasaki commented on MAPREDUCE-6525: - s/similar to MAPREDUCE-6525/similar to MAPREDUCE-6524/ > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6515) Update Application priority in AM side from AM-RM heartbeat
[ https://issues.apache.org/jira/browse/MAPREDUCE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-6515: --- Attachment: 0005-MAPREDUCE-6515.patch Thank you [~jlowe] for pointing out the findbugs warnings. Updating a patch after correcting the same. Thank you. > Update Application priority in AM side from AM-RM heartbeat > --- > > Key: MAPREDUCE-6515 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6515 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-6515.patch, 0002-MAPREDUCE-6515.patch, > 0003-MAPREDUCE-6515.patch, 0004-MAPREDUCE-6515.patch, > 0005-MAPREDUCE-6515.patch > > > After YARN-4170, Application Priority is available via heartbeat call. Update > this information in AM sothat client can fetch this information via JobStatus > (JobReport) call. > This is as per the discussion happened in MAPREDUCE-5870. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated MAPREDUCE-6499: - Attachment: HADOOP-MAPREDUCE-6499.5.patch add missed the elapsed time value. > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976676#comment-14976676 ] Masatake Iwasaki commented on MAPREDUCE-6525: - The failure of TestJobName is not related because it does not use mini cluster. The error message of TestNonExistentJob is similar to MAPREDUCE-6525 and it implies that the fix is not enough. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976793#comment-14976793
]
Hadoop QA commented on MAPREDUCE-6499:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 8s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 28s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 43s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 5m 59s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 45m 55s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12769012/HADOOP-MAPREDUCE-6499.5.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / aa09880 |
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6091/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6091/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6091/console |
This message was automatically generated.
> Add More Informations Of Jobs In JobHistory Main Page
> -
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Trivial
> Attachments: HADOOP-MAPREDUCE-6499.2.patch,
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch,
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.patch
>
>
> Now in JobHistory Main Page show too little information about finished
> jobs,even don't have the job's running total time,only
> startTime,finishedTime.So,in jobHistory main page,we can add more
> informations about jobs, that we can better analyze jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6412: Affects Version/s: (was: 2.7.0) 3.0.0 > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6515) Update Application priority in AM side from AM-RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977055#comment-14977055
]
Hadoop QA commented on MAPREDUCE-6515:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 10m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 43s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 2s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 55s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 10m 46s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 54s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 2m 15s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 6m 38s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 77m 42s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12769016/0005-MAPREDUCE-6515.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / faeb6a3 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6092/console |
This message was automatically generated.
> Update Application priority in AM side from AM-RM heartbeat
> ---
>
> Key: MAPREDUCE-6515
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6515
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-6515.patch, 0002-MAPREDUCE-6515.patch,
> 0003-MAPREDUCE-6515.patch, 0004-MAPREDUCE-6515.patch,
> 0005-MAPREDUCE-6515.patch
>
>
> After YARN-4170, Application Priority is available via heartbeat call. Update
> this information in AM sothat client can fetch this information via JobStatus
> (JobReport) call.
> This is as per the discussion happened in MAPREDUCE-5870.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6520) Migrate MR Client test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated MAPREDUCE-6520: --- Attachment: MAPREDUCE-6520-2.patch v2 of the patch now compiles with trunk and the test cases are running successfully locally. Submitting v2 of the patch for review. > Migrate MR Client test cases > > > Key: MAPREDUCE-6520 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6520 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6520-1.patch, MAPREDUCE-6520-2.patch > > > Migrate the mr client test cases from JUnit3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6520) Migrate MR Client test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated MAPREDUCE-6520: --- Status: Patch Available (was: Open) > Migrate MR Client test cases > > > Key: MAPREDUCE-6520 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6520 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6520-1.patch, MAPREDUCE-6520-2.patch > > > Migrate the mr client test cases from JUnit3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6505) Migrate io test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated MAPREDUCE-6505: --- Status: Patch Available (was: Open) Submitting v2 of the patch that includes whitespace fixes. The previous test failures are unrelated to this patch looks like. > Migrate io test cases > - > > Key: MAPREDUCE-6505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6505 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6505-1.patch, MAPREDUCE-6505-2.patch > > > Migrating just the io test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6391: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) +1 committed to trunk since native task isn't in branch-2 yet. ;) > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6416: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) +1 committed to trunk. Thanks! > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977086#comment-14977086 ] Hudson commented on MAPREDUCE-6412: --- SUCCESS: Integrated in Hadoop-trunk-Commit #8716 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8716/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977090#comment-14977090 ] Hudson commented on MAPREDUCE-6435: --- SUCCESS: Integrated in Hadoop-trunk-Commit #8716 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8716/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc * hadoop-mapreduce-project/CHANGES.txt > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977089#comment-14977089 ] Hudson commented on MAPREDUCE-6391: --- SUCCESS: Integrated in Hadoop-trunk-Commit #8716 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8716/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6412: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) +1 committed to trunk. Thanks! > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6416: Affects Version/s: (was: 2.7.0) 3.0.0 > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977088#comment-14977088 ] Hudson commented on MAPREDUCE-6416: --- SUCCESS: Integrated in Hadoop-trunk-Commit #8716 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8716/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/CHANGES.txt > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6435: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) +1 committed to trunk. Thanks! > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6505) Migrate io test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated MAPREDUCE-6505: --- Attachment: MAPREDUCE-6505-2.patch > Migrate io test cases > - > > Key: MAPREDUCE-6505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6505 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6505-1.patch, MAPREDUCE-6505-2.patch > > > Migrating just the io test cases -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6520) Migrate MR Client test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Cote updated MAPREDUCE-6520: --- Status: Open (was: Patch Available) > Migrate MR Client test cases > > > Key: MAPREDUCE-6520 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6520 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6520-1.patch > > > Migrate the mr client test cases from JUnit3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6520) Migrate MR Client test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977074#comment-14977074 ] Dustin Cote commented on MAPREDUCE-6520: Canceling this patch while I investigate the javac failure. > Migrate MR Client test cases > > > Key: MAPREDUCE-6520 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6520 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Reporter: Dustin Cote >Assignee: Dustin Cote >Priority: Trivial > Attachments: MAPREDUCE-6520-1.patch > > > Migrate the mr client test cases from JUnit3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977313#comment-14977313 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977310#comment-14977310 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/CHANGES.txt > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977315#comment-14977315 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc * hadoop-mapreduce-project/CHANGES.txt > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977314#comment-14977314 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc * hadoop-mapreduce-project/CHANGES.txt fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977401#comment-14977401 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/544/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc * hadoop-mapreduce-project/CHANGES.txt > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977399#comment-14977399 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/544/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977400#comment-14977400 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/544/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc * hadoop-mapreduce-project/CHANGES.txt fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977397#comment-14977397 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/544/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6505) Migrate io test cases
[
https://issues.apache.org/jira/browse/MAPREDUCE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977536#comment-14977536
]
Hadoop QA commented on MAPREDUCE-6505:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 7m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 29 new or modified test files. |
| {color:green}+1{color} | javac | 9m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 23s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 20s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 53s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 8m 29s | Tests passed in
hadoop-common. |
| | | 32m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12769091/MAPREDUCE-6505-2.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 68ce93c |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6093/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6093/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6093/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6093/console |
This message was automatically generated.
> Migrate io test cases
> -
>
> Key: MAPREDUCE-6505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6505
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: test
>Reporter: Dustin Cote
>Assignee: Dustin Cote
>Priority: Trivial
> Attachments: MAPREDUCE-6505-1.patch, MAPREDUCE-6505-2.patch
>
>
> Migrating just the io test cases
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977418#comment-14977418 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #605 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/605/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc * hadoop-mapreduce-project/CHANGES.txt fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977417#comment-14977417 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #605 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/605/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977415#comment-14977415 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #605 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/605/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977419#comment-14977419 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #605 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/605/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977476#comment-14977476 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #592 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/592/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977475#comment-14977475 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #592 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/592/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/CHANGES.txt > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977478#comment-14977478 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #592 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/592/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc * hadoop-mapreduce-project/CHANGES.txt > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977473#comment-14977473 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #592 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/592/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc * hadoop-mapreduce-project/CHANGES.txt > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977378#comment-14977378 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1328 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1328/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977381#comment-14977381 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1328 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1328/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc * hadoop-mapreduce-project/CHANGES.txt fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977380#comment-14977380 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1328 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1328/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977382#comment-14977382 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1328 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1328/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc * hadoop-mapreduce-project/CHANGES.txt > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977445#comment-14977445 ] Lin Yiqun commented on MAPREDUCE-6499: -- Thanks [~ajisakaa] review. I update the patch.But there is a problem when the history job is larger enough, it will be load much more job infos and so slowly to open jobhistory pages. > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6435) MapReduce client assumes the world is x86
[ https://issues.apache.org/jira/browse/MAPREDUCE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977781#comment-14977781 ] Hudson commented on MAPREDUCE-6435: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2482 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2482/]) MAPREDUCE-6435. MapReduce client assumes the world is x86 (Alan Burlison (aw: rev 68ce93c32e1cf4344929b26597780ec387efa107) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > MapReduce client assumes the world is x86 > - > > Key: MAPREDUCE-6435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6435 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 2.7.1 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6435.001.patch > > > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > contains x86-only code and assumes it's going to work by default. The logic > should be inverted so that it only uses the x86 code when built on x86 and > otherwise defaults to the architecture-independent code. That will prevent > changes having to be made for each new architecture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977612#comment-14977612
]
Akira AJISAKA commented on MAPREDUCE-6499:
--
Thank you for updating the patch, [~linyiqun].
bq. But there is a problem when the history job is larger enough, it will be
load much more job infos and so slowly to open jobhistory pages.
Good point. If you really need to sort by average map time and other
information, we need to add them into index not to load full job information
(See JobIndexInfo.java). In addition, we need to choose what information to add
very carefully considering performance.
IMO, adding elapsed time to the main page is a good idea because
* Elapsed time can be calculated without loading full job info or editing index
file.
* Users can easily detect long-running-jobs by sorting by elapsed time
{code}
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Job
ID")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Name")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("User")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Queue")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("State")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Maps
Total")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Maps
Completed")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Reduces
Total")._()._().
-
th().input("search_init").$type(InputType.text).$name("start_time").$value("Reduces
Completed")._()._()
(snip)
+th().input("search_init").$type(InputType.text)
+ .$name("job_id").$value("Job ID")._()._().
+th().input("search_init").$type(InputType.text)
+ .$name("name").$value("Name")._()._().
+th().input("search_init").$type(InputType.text)
+ .$name("user").$value("User")._()._().
+th().input("search_init").$type(InputType.text)
+ .$name("queue").$value("Queue")._()._().
+th().input("search_init").$type(InputType.text)
+ .$name("state").$value("State")._()._().
+th().input("search_init").$type(InputType.text)
+ .$name("maps_total").$value("Maps Total")._()._().
+th().input("search_init").$type(InputType.text).
+ $name("maps_completed").$value("Maps Completed")._()._().
+th().input("search_init").$type(InputType.text).
+ $name("reduces_total").$value("Reduces Total")._()._().
+th().input("search_init").$type(InputType.text).
+ $name("reduces_completed").$value("Reduces Completed")._()._().
{code}
Good catch. We need to fix it.
> Add More Informations Of Jobs In JobHistory Main Page
> -
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Trivial
> Attachments: HADOOP-MAPREDUCE-6499.2.patch,
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch,
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.patch
>
>
> Now in JobHistory Main Page show too little information about finished
> jobs,even don't have the job's running total time,only
> startTime,finishedTime.So,in jobHistory main page,we can add more
> informations about jobs, that we can better analyze jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6416) Not all platforms have d_type in struct dirent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1499#comment-1499 ] Hudson commented on MAPREDUCE-6416: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2482 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2482/]) MAPREDUCE-6416. Not all platforms have d_type in struct dirent (Alan (aw: rev 5c24fe7f91970dedae35906ff7990ea5410f070e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc * hadoop-mapreduce-project/CHANGES.txt > Not all platforms have d_type in struct dirent > -- > > Key: MAPREDUCE-6416 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6416 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: Solaris >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6416.001.patch > > > list() in > hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/FileSystem.cc > assumes that struct dirent always contains a d_type field which can be used > to determine the type of the directory entry. This field is non-POSIX and is > not available on all platforms. On platforms where it is not available, > stat() should be used instead to determine the type of the directory entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6391) util/Timer.cc completely misunderstands _POSIX_CPUTIME
[ https://issues.apache.org/jira/browse/MAPREDUCE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977780#comment-14977780 ] Hudson commented on MAPREDUCE-6391: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2482 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2482/]) MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME (aw: rev bcc4c746a88cc698b2937b85ec845b9d8cc736d2) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Timer.cc fix changes.txt for MAPREDUCE-6391. (aw: rev 0e344902aaca417813c7191395cd5c73dfcd4220) * hadoop-mapreduce-project/CHANGES.txt > util/Timer.cc completely misunderstands _POSIX_CPUTIME > -- > > Key: MAPREDUCE-6391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6391 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6391.001.patch > > > util/Timer.cc contains the following code: > clock_gettime(_POSIX_CPUTIME, ); > I'm not exactly sure what sort of time value this code is attempting to > obtain (the process's on-CPU time perhaps?) but it is completely wrong, even > on Linux. _POSIX_CPUTIME is a feature test macro and is *not* a valid > parameter to clock_gettime. From the clock_gettime manpage: > bq. The symbols _POSIX_MONOTONIC_CLOCK, _POSIX_CPUTIME, _POSIX_THREAD_CPUTIME > indicate that CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID, > CLOCK_THREAD_CPUTIME_ID are available > On my Linux distro the value of _POSIX_CPUTIME is defined as 0. That > corresponds to > bq. #define __CLOCK_REALTIME0 0 /* obsolete; same as > CLOCK_REALTIME */ > And CLOCK_REALTIME is > bq. #define CLOCK_REALTIME 3 /* wall clock */ > Which I suspect is not what is required here. > Getting cross-platform CPU time is fiddly, the following blog post has a good > summary: > http://nadeausoftware.com/articles/2012/03/c_c_tip_how_measure_cpu_time_benchmarking > (code examples are CC licensed so should be no taint issues) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6412) Make hadoop-mapreduce-client Native code -Wall-clean
[ https://issues.apache.org/jira/browse/MAPREDUCE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1498#comment-1498 ] Hudson commented on MAPREDUCE-6412: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2482 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2482/]) MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean (aw: rev ab99d953e464f240691b6eb45c28f2f4db27d8f0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Random.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestPrimitives.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MemoryBlock.cc * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/Buffers.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/WritableUtils.cc > Make hadoop-mapreduce-client Native code -Wall-clean > > > Key: MAPREDUCE-6412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6412 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 > Environment: As we specify -Wall as a default compilation flag, it > would be helpful if the Native code was -Wall-clean >Reporter: Alan Burlison >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: MAPREDUCE-6412.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
