[jira] [Created] (MAPREDUCE-5766) Ping messages from attempts should be moved to DEBUG
Ramya Sunil created MAPREDUCE-5766: -- Summary: Ping messages from attempts should be moved to DEBUG Key: MAPREDUCE-5766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5766 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Ramya Sunil Priority: Minor Fix For: 0.24.0 Messages such as org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from attempt_1391416522080_0015_m_00_0 in AM logs should be moved to DEBUG. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-858) NPE in heartbeat if mapred.job.tracker.history.completed.location is not writable
[
https://issues.apache.org/jira/browse/MAPREDUCE-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramya Sunil resolved MAPREDUCE-858.
---
Resolution: Fixed
This is no longer an issue. JT gracefully shuts down if
mapred.job.tracker.history.completed.location is not writable.
NPE in heartbeat if mapred.job.tracker.history.completed.location is not
writable
Key: MAPREDUCE-858
URL: https://issues.apache.org/jira/browse/MAPREDUCE-858
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.1, 0.21.0
Reporter: Ramya Sunil
Priority: Minor
If mapred.job.tracker.history.completed.location has been configured to
write to a location which is not writable by JT, NullPointerException is
thrown in TT heartbeat. Below is the Exception obtained:
{noformat}
2009-08-13 07:56:02,815 INFO org.apache.hadoop.ipc.Server: IPC Server handler
handler on port, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1e7a6ae, false, false,
true, 1775) from ip:port: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager.moveToDone(JobHistory.java:215)
at
org.apache.hadoop.mapred.JobHistory$JobInfo.markCompleted(JobHistory.java:1071)
at
org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2413)
at
org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2729)
at
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2327)
at
org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2259)
at
org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:957)
at
org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3946)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3123)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2861)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
{noformat}
Instead of an NPE, it would be helpful if an useful error message is logged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3080) dfs calls from streaming fails with ExceptionInInitializerError
dfs calls from streaming fails with ExceptionInInitializerError
---
Key: MAPREDUCE-3080
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3080
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0
Reporter: Ramya Sunil
Fix For: 0.20.205.0
Dfs calls from streaming seem to fail with the following error:
{noformat}
Exception in thread main java.lang.ExceptionInInitializerError
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
Caused by: org.apache.commons.logging.LogConfigurationException: User-specified
log class 'org.apache.commons.logging.impl.Log4JLogger' cannot be found or is
not useable.
at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:874)
at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
at org.apache.hadoop.conf.Configuration.clinit(Configuration.java:142)
... 3 more
{noformat}
commons-logging-1.1.1.jar is in the classpath. An easy way to reproduce this
is, on a secure deploy, hadoop --config $HADOOP_CONF_DIR jar
hadoop-streaming.jar -input UserInput -output Out -mapper hadoop --config
$HADOOP_CONF_DIR dfs -help -reducer NONE
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3063) Mapreduce trunk Commit builds are failing
Mapreduce trunk Commit builds are failing - Key: MAPREDUCE-3063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3063 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Ramya Sunil Priority: Critical Mapreduce trunk commit builds are failing due to test failures. See https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/946/testReport/ for more details. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112173#comment-13112173 ] Ramya Sunil commented on MAPREDUCE-2789: Eric, The output looks good and has all the necessary details. Just one more thing, the number of containers does not give the break up of how many are used by each of the map and reduce task i.e. information such as 3 map tasks running using 3 containers and 1 reduce task running using 1 container. This information is very useful for High RAM jobs. E.g. for a High RAM job which has 3 map tasks, each requiring 2 containers and 1 reduce task requiring 1 container, displaying the number of containers as 7(6+1) would not convey the number of containers actually used by each task. Perhaps something like Map Containers and Reduce containers would be useful. Thoughts? [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3051) HADOOP_CONF_DIR exported twice in the classpath
HADOOP_CONF_DIR exported twice in the classpath --- Key: MAPREDUCE-3051 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3051 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 HADOOP_CONF_DIR is exported twice in the classpath during RM, NM and container startup time. Not an issue so far but seems redundant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2763) IllegalArgumentException while using the dist cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-2763. Resolution: Fixed I see this issue being fixed in the latest code base. Hence resolving this. IllegalArgumentException while using the dist cache --- Key: MAPREDUCE-2763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2763 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 IllegalArgumentException is seen while using distributed cache to cache some files and custom jars in classpath. A simple way to reproduce this error is by using a streaming job: hadoop jar hadoop-streaming.jar -libjars file://path to custom jar -input path to input file -output out -mapper cat -reducer NONE -cacheFile hdfs://path to some file#linkname This is a regression introduced and the same command works fine on 0.20.x -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3043) Missing containers info on the nodes page
Missing containers info on the nodes page - Key: MAPREDUCE-3043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3043 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The containers info on the nodes page on the RM seems to be missing. This was useful in understanding the usage on each of the nodemanagers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3044) Pipes jobs stuck without making progress
Pipes jobs stuck without making progress Key: MAPREDUCE-3044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3044 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 A simple example pipes job gets stuck without making any progress. The AM is launched but the maps do not make any progress. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
Elapsed time filter on jobhistory server displays incorrect table entries - Key: MAPREDUCE-3045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 The elapsed time filter on the jobhistory server filters incorrect information. For e.g. on a cluster where the elapsed time of all the tasks is either 7 or 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2790) [MR-279] Add additional field for storing the AM/job history info on CLI
[
https://issues.apache.org/jira/browse/MAPREDUCE-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103200#comment-13103200
]
Ramya Sunil commented on MAPREDUCE-2790:
Just to clarify once again, in 0.20.x we have a field SchedulingInfo having
information such as the following:
{noformat} 2 running map tasks using 4 map slots. 0 additional slots reserved.
10 running reduce tasks using 20 reduce slots. 0 additional slots reserved.
{noformat}
This information is missing in branch-23 and has to be added.
Also, in branch-23, the AM information/job history location is being
incorrectly displayed under the SchedulingInfo field. As Arun pointed out in
his first comment, an additional field to display the URI of either AM Web-UI
for running apps or for JobHistory URI for completed apps has to be added.
[MR-279] Add additional field for storing the AM/job history info on CLI
Key: MAPREDUCE-2790
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2790
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Ravi Prakash
Fix For: 0.23.0
Attachments: MAPREDUCE-2790.v1.txt, MAPREDUCE-2790.v2.txt
bin/mapred job [-list [all]] displays the AM or job history location in the
SchedulingInfo field. An additional column has to be added to display the
AM/job history information. Currently, the output reads:
{noformat}
JobId State StartTime UserNameQueue Priority
SchedulingInfo
jobID FAILED 0 ramya default NORMAL AM
information/job history location
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2690) Construct the web page for default scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087304#comment-13087304 ] Ramya Sunil commented on MAPREDUCE-2690: General scheduler information such as configured queue names and a handle to apps specific information such as apps running, user, job name, state of the app, progress, tracking UI etc would be useful to have. Construct the web page for default scheduler Key: MAPREDUCE-2690 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0 Currently, the web page for default scheduler reads as Under construction. This is a long known issue, but could not find a tracking ticket. Hence opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[MR-279] Node entries on the RM UI are not sortable --- Key: MAPREDUCE-2843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The nodemanager entries on the RM UI is not sortable unlike the other web pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2844) [MR-279] Incorrect node ID info
[MR-279] Incorrect node ID info Key: MAPREDUCE-2844 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2844 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The node ID info for the nodemanager entires on the RM UI incorrectly displays the value of $yarn.server.nodemanager.address instead of the ID. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2834) [MR-279] Enable dense update for file sink metrics
[MR-279] Enable dense update for file sink metrics -- Key: MAPREDUCE-2834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, if File sink is enabled for MRAppMaster or Resourcemanager, it does not populate the file with all the available attributes. It would be useful for debugging and admin purpose to have all the metrics populated in the file. For eg: MRAppMaster metrics currently logs value only for JobsRunning even though the total available job level metrics are JobsCompleted, JobsFailed, JobsKilled, JobsPreparing etc -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2799) [MR-279] NPE is throwing on job -status Invalid Job ID/Job Id doesn't exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramya Sunil resolved MAPREDUCE-2799.
Resolution: Duplicate
Devaraj, there is already a known bug MAPREDUCE-2686 for the same. Hence
resolving this as duplicate.
[MR-279] NPE is throwing on job -status Invalid Job ID/Job Id doesn't exist
-
Key: MAPREDUCE-2799
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2799
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
{code:xml}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:113)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:101)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:94)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:384)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:223)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2800) clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files
clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files - Key: MAPREDUCE-2800 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2800 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 for all the map tasks for the last 4 progress interval in the jobhistory files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2801) Include the native libs in java.library.path
Include the native libs in java.library.path - Key: MAPREDUCE-2801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to path to native libs:$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[MR-279] Jobhistory filenames should have jobID to help in better parsing -- Key: MAPREDUCE-2802 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 For jobID such as job_1312933838300_0007, jobhistory file names are named as job%5F1312933838300%5F0007_submit_time_ramya_jobname_finish_time_1_1_SUCCEEDED.jhist It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2790) [MR-279] Add additional field for storing the AM/job history info on CLI
[MR-279] Add additional field for storing the AM/job history info on CLI
Key: MAPREDUCE-2790
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2790
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
bin/mapred job [-list [all]] displays the AM or job history location in the
SchedulingInfo field. An additional column has to be added to display the
AM/job history information. Currently, the output reads:
{noformat}
JobId State StartTime UserNameQueue Priority
SchedulingInfo
jobID FAILED 0 ramya default NORMAL AM
information/job history location
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[MR-279] Missing/incorrect info on job -status CLI --- Key: MAPREDUCE-2791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 There are a couple of details missing/incorrect on the job -status command line output for completed jobs: 1. Incorrect job file 2. map() completion is always 0 3. reduce() completion is always set to 0 4. history URL is empty 5. Missing launched map tasks 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2792) [MR-279] Replace IP addresses with hostnames
[MR-279] Replace IP addresses with hostnames Key: MAPREDUCE-2792 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2792 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, all the logs, UI, CLI have IP addresses of the NM/RM, which are difficult to manage. It will be useful to have hostnames like in 0.20.x for easier debugging and maintenance purpose. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2793) [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs
[MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs -- Key: MAPREDUCE-2793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2793 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 appIDs, jobIDs and attempt/container ids are not consistently named in the logs, console and UI. For consistency purpose, they all have to follow a common naming convention. Currently, For appID = On the RM UI: app_1308259676864_5 On the JHS UI: No appID Console/logs: No appID mapred-local dirs are named as: application_1308259676864_0005 For jobID = On the RM UI: job_1308259676864_5_5 JHS UI: job_1308259676864_5_5 Console/logs: job_1308259676864_0005 mapred-local dirs are named as: No jobID For attemptID On the RM UI: attempt_1308259676864_5_5_m_24_0 JHS attempt_1308259676864_5_5_m_24_0 Console/logs: attempt_1308259676864_0005_m_24_0 mapred-local dirs are named as: container_1308259676864_0005_24 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2794) [MR-279] Incorrect metrics value for AvailableGB per queue per user
[MR-279] Incorrect metrics value for AvailableGB per queue per user --- Key: MAPREDUCE-2794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 AvailableGB per queue is not the same as AvailableGB per queue per user when the user limit is set to 100%. i.e. if the total available GB of the cluster is 60, and queue default has 92% capacity with 100% as the user limit, AvailableGB per queue default = 55 (i.e. 0.92*60) whereas AvailableGB per queue for user ramya is 56 (however it should be 55 = 0.92*60*1) Also, unlike the AvailableGB/queue, AvailableGB/queue/user is not decremented when user ramya is running apps on the default queue. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2795) [MR-279] AppsKilled is never incremented
[MR-279] AppsKilled is never incremented Key: MAPREDUCE-2795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2795 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 AppsKilled metrics is never incremented even though there are killed jobs in the system. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2796) [MR-279] Start time for all the apps is set to 0
[MR-279] Start time for all the apps is set to 0 Key: MAPREDUCE-2796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2796 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The start time for all the apps in the output of job -list is set to 0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2774) Add a startup msg while starting RM/NM
Add a startup msg while starting RM/NM -- Key: MAPREDUCE-2774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Add a startup msg while starting NM/RM indicating the version, build details etc. This will help in easier parsing of logs and debugging. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2774) [MR-279] Add a startup msg while starting RM/NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil updated MAPREDUCE-2774: --- Summary: [MR-279] Add a startup msg while starting RM/NM (was: Add a startup msg while starting RM/NM) [MR-279] Add a startup msg while starting RM/NM --- Key: MAPREDUCE-2774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Add a startup msg while starting NM/RM indicating the version, build details etc. This will help in easier parsing of logs and debugging. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[MR-279] Decommissioned node does not shutdown -- Key: MAPREDUCE-2775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 A Nodemanager which is decommissioned by an admin via refreshnodes does not automatically shutdown. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2766) [MR-279] Set correct permissions for files in dist cache
[MR-279] Set correct permissions for files in dist cache Key: MAPREDUCE-2766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2766 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, the files in both public and private dist cache are having 777 permission. Also, the group ownership of files on private cache have to be set to $TT_SPECIAL_GROUP -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2768) [MR-279] NMs not being blacklisted as determined by health scripts
[MR-279] NMs not being blacklisted as determined by health scripts -- Key: MAPREDUCE-2768 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2768 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The NMs are not being blacklisted via the node health script. Below is the configuration used: yarn.server.nodemanager.healthchecker.script.path=path to node health script which blacklists a NM yarn.server.nodemanager.healthchecker.interval=10 yarn.server.nodemanager.healthchecker.script.timeout=12 The node continues to be healthy forever. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir after job completion
[MR-279] - Cleanup staging dir after job completion
---
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
The files created under the staging dir have to be deleted after job
completion. Currently, all job.* files remain forever in the
${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2763) IllegalArgumentException while using the dist cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073835#comment-13073835
]
Ramya Sunil commented on MAPREDUCE-2763:
Below is the complete stacktrace:
{noformat}
Exception in thread main java.lang.IllegalArgumentException: Invalid
specification for distributed-cache artifacts of type FILE : #uris=1
#timestamps=2 #visibilities=2
at
org.apache.hadoop.mapred.YARNRunner.parseDistributedCacheArtifacts(YARNRunner.java:411)
at
org.apache.hadoop.mapred.YARNRunner.setupDistributedCache(YARNRunner.java:392)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:234)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:388)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1064)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1061)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1061)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
at
org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1010)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:133)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at
org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
{noformat}
IllegalArgumentException while using the dist cache
---
Key: MAPREDUCE-2763
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2763
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
IllegalArgumentException is seen while using distributed cache to cache some
files and custom jars in classpath.
A simple way to reproduce this error is by using a streaming job:
hadoop jar hadoop-streaming.jar -libjars file://path to custom jar -input
path to input file -output out -mapper cat -reducer NONE -cacheFile
hdfs://path to some file#linkname
This is a regression introduced and the same command works fine on 0.20.x
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2763) IllegalArgumentException while using the dist cache
IllegalArgumentException while using the dist cache --- Key: MAPREDUCE-2763 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2763 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 IllegalArgumentException is seen while using distributed cache to cache some files and custom jars in classpath. A simple way to reproduce this error is by using a streaming job: hadoop jar hadoop-streaming.jar -libjars file://path to custom jar -input path to input file -output out -mapper cat -reducer NONE -cacheFile hdfs://path to some file#linkname This is a regression introduced and the same command works fine on 0.20.x -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2755) MR-279: AM writes logs to stderr
MR-279: AM writes logs to stderr Key: MAPREDUCE-2755 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2755 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently the AM logs are written to $YARN_LOG_DIR/appID/containerID/stderr. In order to maintain consistency with other container logs, it probably should be moved to syslog. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2757) [MR-279] Redundant file: directory created in appcache
[MR-279] Redundant file: directory created in appcache
-
Key: MAPREDUCE-2757
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Priority: Minor
Fix For: 0.23.0
A redundant directory called file: is being created under
${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which
further has a directory structure
${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache
which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2758) MR-279: Log info about the location of dist cache
MR-279: Log info about the location of dist cache - Key: MAPREDUCE-2758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2758 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 Currently, there is no log info available about the actual location of the file/archive in dist cache being used by the task except for the ln command in task.sh. We need to log this information to help in debugging esp in those cases where there are more than one archive with the same name. In 0.20.x, in task logs, one could find log info such as the following: INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: distcache location/archive - mapred.local.dir/archive -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2737) Update the progress of jobs on client side
Update the progress of jobs on client side -- Key: MAPREDUCE-2737 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2737 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 The progress of the jobs are not being correctly updated on the client side. The map progress halts at 66% and both map/reduce progress % does not display 100 when the job completes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2737) Update the progress of jobs on client side
[
https://issues.apache.org/jira/browse/MAPREDUCE-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071907#comment-13071907
]
Ramya Sunil commented on MAPREDUCE-2737:
The progress of a sample job on 0.20.x is being displayed as:
{noformat}
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: map 10% reduce 0%
INFO mapred.JobClient: map 70% reduce 0%
INFO mapred.JobClient: map 90% reduce 0%
INFO mapred.JobClient: map 100% reduce 5%
INFO mapred.JobClient: map 100% reduce 10%
INFO mapred.JobClient: map 100% reduce 14%
INFO mapred.JobClient: map 100% reduce 35%
INFO mapred.JobClient: map 100% reduce 42%
INFO mapred.JobClient: map 100% reduce 49%
INFO mapred.JobClient: map 100% reduce 59%
INFO mapred.JobClient: map 100% reduce 73%
INFO mapred.JobClient: map 100% reduce 83%
INFO mapred.JobClient: map 100% reduce 93%
INFO mapred.JobClient: map 100% reduce 100%
INFO mapred.JobClient: Job complete: job_201105270127_0021
{noformat}
Whereas the same job progress on mrv2 is being displayed as:
{noformat}
INFO mapreduce.Job: map 0% reduce 0%
INFO mapreduce.Job: map 60% reduce 0%
INFO mapreduce.Job: map 66% reduce 12%
INFO mapreduce.Job: map 66% reduce 32%
INFO mapreduce.Job: Job job_1311726209366_0214 completed successfully
{noformat}
Update the progress of jobs on client side
--
Key: MAPREDUCE-2737
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2737
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
The progress of the jobs are not being correctly updated on the client side.
The map progress halts at 66% and both map/reduce progress % does not display
100 when the job completes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2738) Missing cluster level stats on the RM UI
Missing cluster level stats on the RM UI Key: MAPREDUCE-2738 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Cluster usage information such as the following are currently not available in the RM UI. - Total number of apps submitted so far - Total number of containers running/total memory usage - Total capacity of the cluster (in terms of memory) - Reserved memory - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward - Excluded NMs - Handle to the jobhistory server -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1986) History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path
[ https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramya Sunil resolved MAPREDUCE-1986. Resolution: Duplicate This issue is exactly the same as MAPREDUCE-2463. Since there is already a patch available in MAPREDUCE-2463, resolving this as duplicate. Thanks Bhallamudi. History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path --- Key: MAPREDUCE-1986 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Ramya Sunil When hadoop.job.history.location is configured to a HDFS path and when the job retires, the history files are never moved to mapred.job.tracker.history.completed.location and they remain forever in hadoop.job.history.location . The jobhistory is never viewable thereafter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2687) Non superusers unable to launch apps in a non-secure cluster
Non superusers unable to launch apps in a non-secure cluster Key: MAPREDUCE-2687 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2687 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Apps of non superuser fail to succeed in a non-secure environment. Only the superuser(i.e. one who started/owns the mrv2 cluster) is able to launch apps successfully. However, when a normal user submits a job, the job fails. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2687) Non superusers unable to launch apps in a non-secure cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066085#comment-13066085
]
Ramya Sunil commented on MAPREDUCE-2687:
The job is submitted as user ramya and mr_superuser is the owner of the mrv2
cluster. Below is the snippet of the logs:
{noformat}
INFO impl.JobImpl: DEBUG --- startJobs: parent=/user/mr_superuser/.staging
child=jobID
INFO impl.JobImpl: Adding job token for jobID to jobTokenSecretManager
INFO impl.JobImpl: Writing back the job-token file on the remote file
system:hdfs://namenodehost:port/user/mr_superuser/.staging/jobID/appTokens
WARN impl.JobImpl: Job init failed
org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File does
not exist:
hdfs://namenodehost:port/user/mr_superuser/.staging/jobID/job.splitmetainfo
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1071)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:800)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:771)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:669)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:116)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:508)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:240)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:558)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://namenodehost:port/user/mr_superuser/.staging/jobID/job.splitmetainfo
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:739)
at
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:50)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1066)
... 11 more
11/07/14 18:32:17 ERROR app.MRAppMaster: Caught throwable. Exiting:
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.abortJob(JobImpl.java:1118)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$2900(JobImpl.java:116)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:947)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:771)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:669)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:116)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:508)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:240)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:558)
{noformat}
Non superusers unable to launch apps in a non-secure cluster
Key: MAPREDUCE-2687
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2687
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
Apps of non superuser fail to succeed in a non-secure environment. Only the
superuser(i.e. one who started/owns the mrv2 cluster) is able to launch apps
successfully. However, when a normal user submits a job, the job fails.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2689) InvalidStateTransisiton when AM is not assigned to a job
InvalidStateTransisiton when AM is not assigned to a job Key: MAPREDUCE-2689 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2689 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 In cases where an AM is not being assigned to a job, RELEASED at COMPLETED invalid event is observed. This is easily reproducible in cases such as MAPREDUCE-2687. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2689) InvalidStateTransisiton when AM is not assigned to a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066109#comment-13066109
]
Ramya Sunil commented on MAPREDUCE-2689:
Below is the complete stacktrace:
{noformat}
ERROR
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.ApplicationImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
RELEASED at COMPLETED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:297)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
at
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.ApplicationImpl.handle(ApplicationImpl.java:587)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:202)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:187)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:111)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
InvalidStateTransisiton when AM is not assigned to a job
Key: MAPREDUCE-2689
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2689
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
In cases where an AM is not being assigned to a job, RELEASED at COMPLETED
invalid event is observed. This is easily reproducible in cases such as
MAPREDUCE-2687.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2690) Construct the web page for default scheduler
Construct the web page for default scheduler Key: MAPREDUCE-2690 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, the web page for default scheduler reads as Under construction. This is a long known issue, but could not find a tracking ticket. Hence opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2686) NPE while requesting info for a non-existing job
NPE while requesting info for a non-existing job Key: MAPREDUCE-2686 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2686 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 While performing job related operations such as job -kill, -status, -events etc for an unknown job, the following NPE is seen: Exception in thread main java.lang.NullPointerException at org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:112) at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:100) at org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:93) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154) at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:254) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2677) MR-279: 404 error while accessing pages from history server
MR-279: 404 error while accessing pages from history server --- Key: MAPREDUCE-2677 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2677 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Accessing the following pages from the history server, causes 404 HTTP error 1. Cluster- About 2. Cluster - Applications 3. Cluster - Scheduler 4. Application - About -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
