[jira] [Commented] (MAPREDUCE-6944) MR job got hanged forever when some NMs unstable for some time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397543#comment-16397543
]
Ashwin Shankar commented on MAPREDUCE-6944:
---
[~daemon] we are having similar problems. Were you able to root cause this?
> MR job got hanged forever when some NMs unstable for some time
> --
>
> Key: MAPREDUCE-6944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, resourcemanager
>Reporter: YunFan Zhou
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> We encountered several jobs in the production environment due to the fact
> that some of the NM unstable cause one *MAP* of the job to be stuck there,
> and the job can't finish properly.
> However, the problems we encountered were different from those mentioned in
> [https://issues.apache.org/jira/browse/MAPREDUCE-6513]. Because in our
> scenario, all of *MR REDUCEs* does not start executing.
> But when I manually kill the hanged *MAP*, the job will be finished normally.
> {noformat}
> 2017-08-17 12:25:06,548 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start
> threshold not met. completedMapsForReduceSlowstart 15564
> 2017-08-17 12:25:07,555 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e84_1502793246072_73922_01_015700
> 2017-08-17 12:25:07,556 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=
> 2017-08-17 12:25:07,556 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start
> threshold not met. completedMapsForReduceSlowstart 15564
> 2017-08-17 12:25:07,556 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:1009 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:15563 CompletedReds:0 ContAlloc:15723 ContRel:26
> HostLocal:4575 RackLocal:8121
> {noformat}
> {noformat}
> 2017-08-17 14:49:41,793 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1009 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:15563 CompletedReds:0 ContAlloc:15724 ContRel:26
> HostLocal:4575 RackLocal:8121
> 2017-08-17 14:49:41,794 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Applying ask
> limit of 1 for priority:5 and capability:
> 2017-08-17 14:49:41,799 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1502793246072_73922: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=4236
> 2017-08-17 14:49:41,799 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=
> 2017-08-17 14:49:41,799 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start
> threshold not met. completedMapsForReduceSlowstart 15564
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning
> container Container: [ContainerId:
> container_e84_1502793246072_73922_01_015726, NodeId:
> bigdata-hdp-apache1960.xg01.diditaxi.com:8041, NodeHttpAddress:
> bigdata-hdp-apache1960.xg01.diditaxi.com:8042, Resource: vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service:
> 10.93.111.36:8041 }, ] to fast fail map
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from
> earlierFailedMaps
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_e84_1502793246072_73922_01_015726 to
> attempt_1502793246072_73922_m_012103_5
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=
> 2017-08-17 14:49:42,805 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start
> threshold
[jira] [Commented] (MAPREDUCE-6107) Job history server becomes unresponsive due to stuck thread in epollWait
[
https://issues.apache.org/jira/browse/MAPREDUCE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317121#comment-14317121
]
Ashwin Shankar commented on MAPREDUCE-6107:
---
[~suma.shivaprasad], we havent deployed the image with HDFS-7005 yet, so I
can't comment. Also we haven't seen this issue come up again in the image
without the fix.
Job history server becomes unresponsive due to stuck thread in epollWait
Key: MAPREDUCE-6107
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6107
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.4.0
Reporter: Ashwin Shankar
Attachments: jstack.log
About once every week, we see job history server becomes unresponsive on one
of our 2000 node hadoop cluster. Looking at the thread dump, I see that
multiple threads are blocked on locks acquired by couple of threads, which in
turn are endlessly stuck in epollWait while talking to hdfs to get a history
file.
When the number of blocked threads touches the thread pool size, JHS becomes
unresponsive to new clients requests.
Thread dump attached.
Has anyone seen this before ?
Here is the thread stuck at epollWait.
{code}
IPC Server handler 4 on 10020 daemon prio=10 tid=0x7f7eb10f5000
nid=0x144d runnable [0x7f7e9108d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked 0x0006c89d3240 (a sun.nio.ch.Util$2)
- locked 0x0006c89d3228 (a
java.util.Collections$UnmodifiableSet)
- locked 0x0006bb32f8b8 (a sun.nio.ch.EPollSelectorImpl)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6107) Job history server becomes unresponsive due to stuck thread in epollWait
Ashwin Shankar created MAPREDUCE-6107:
-
Summary: Job history server becomes unresponsive due to stuck
thread in epollWait
Key: MAPREDUCE-6107
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6107
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.4.0
Reporter: Ashwin Shankar
About once every week, we see job history server becomes unresponsive on one of
our 2000 node hadoop cluster. Looking at the thread dump, I see that multiple
threads are blocked on locks acquired by couple of threads, which in turn are
endlessly stuck in epollWait while talking to hdfs to get a history file.
When the number of blocked threads touches the thread pool size, JHS becomes
unresponsive to new clients requests.
Thread dump attached.
Has anyone seen this before ?
Here is the thread stuck at epollWait.
{code}
IPC Server handler 4 on 10020 daemon prio=10 tid=0x7f7eb10f5000
nid=0x144d runnable [0x7f7e9108d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked 0x0006c89d3240 (a sun.nio.ch.Util$2)
- locked 0x0006c89d3228 (a java.util.Collections$UnmodifiableSet)
- locked 0x0006bb32f8b8 (a sun.nio.ch.EPollSelectorImpl)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6107) Job history server becomes unresponsive due to stuck thread in epollWait
[
https://issues.apache.org/jira/browse/MAPREDUCE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-6107:
--
Attachment: jstack.log
Job history server becomes unresponsive due to stuck thread in epollWait
Key: MAPREDUCE-6107
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6107
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.4.0
Reporter: Ashwin Shankar
Attachments: jstack.log
About once every week, we see job history server becomes unresponsive on one
of our 2000 node hadoop cluster. Looking at the thread dump, I see that
multiple threads are blocked on locks acquired by couple of threads, which in
turn are endlessly stuck in epollWait while talking to hdfs to get a history
file.
When the number of blocked threads touches the thread pool size, JHS becomes
unresponsive to new clients requests.
Thread dump attached.
Has anyone seen this before ?
Here is the thread stuck at epollWait.
{code}
IPC Server handler 4 on 10020 daemon prio=10 tid=0x7f7eb10f5000
nid=0x144d runnable [0x7f7e9108d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked 0x0006c89d3240 (a sun.nio.ch.Util$2)
- locked 0x0006c89d3228 (a
java.util.Collections$UnmodifiableSet)
- locked 0x0006bb32f8b8 (a sun.nio.ch.EPollSelectorImpl)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6107) Job history server becomes unresponsive due to stuck thread in epollWait
[
https://issues.apache.org/jira/browse/MAPREDUCE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146846#comment-14146846
]
Ashwin Shankar commented on MAPREDUCE-6107:
---
Thanks Jason ! We'll try out HDFS-7005 and see if the problem persists.
Job history server becomes unresponsive due to stuck thread in epollWait
Key: MAPREDUCE-6107
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6107
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.4.0
Reporter: Ashwin Shankar
Attachments: jstack.log
About once every week, we see job history server becomes unresponsive on one
of our 2000 node hadoop cluster. Looking at the thread dump, I see that
multiple threads are blocked on locks acquired by couple of threads, which in
turn are endlessly stuck in epollWait while talking to hdfs to get a history
file.
When the number of blocked threads touches the thread pool size, JHS becomes
unresponsive to new clients requests.
Thread dump attached.
Has anyone seen this before ?
Here is the thread stuck at epollWait.
{code}
IPC Server handler 4 on 10020 daemon prio=10 tid=0x7f7eb10f5000
nid=0x144d runnable [0x7f7e9108d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked 0x0006c89d3240 (a sun.nio.ch.Util$2)
- locked 0x0006c89d3228 (a
java.util.Collections$UnmodifiableSet)
- locked 0x0006bb32f8b8 (a sun.nio.ch.EPollSelectorImpl)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5956) MapReduce AM should not use maxAttempts to determine if this is the last retry
[
https://issues.apache.org/jira/browse/MAPREDUCE-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106414#comment-14106414
]
Ashwin Shankar commented on MAPREDUCE-5956:
---
[~zjshen], could you please elaborate when and why this fix will lead to
undeleted staging dirs ?
In such cases, would the job go missing on the history server ? Any other
impact ?
I'm asking this, because we are backporting this patch into our release.
MapReduce AM should not use maxAttempts to determine if this is the last retry
--
Key: MAPREDUCE-5956
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5956
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Affects Versions: 2.4.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Priority: Blocker
Fix For: 2.6.0
Attachments: MR-5956.patch, MR-5956.patch
Found this while reviewing YARN-2074. The problem is that after YARN-2074, we
don't count AM preemption towards AM failures on RM side, but MapReduce AM
itself checks the attempt id against the max-attempt count to determine if
this is the last attempt.
{code}
public void computeIsLastAMRetry() {
isLastAMRetry = appAttemptID.getAttemptId() = maxAppAttempts;
}
{code}
This causes issues w.r.t deletion of staging directory etc..
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053913#comment-14053913 ] Ashwin Shankar commented on MAPREDUCE-5418: --- [~kasha], shouldn't this be marked as duplicate rather than fixed ? JobHistoryServer has no information about applications if the MR-AM crashes --- Key: MAPREDUCE-5418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am Affects Versions: 2.0.5-alpha Reporter: Karthik Kambatla Assignee: Robert Kanter Priority: Critical Attachments: JHS-web-ui.png, RM-web-ui.png Currently, the AM writes the job-specific information to HDFS only after it finishes; the JHS needs this info to display anything. If the AM fails, this info is not written and the JHS fails to display anything for that job. While JHS on top of AHS might address this issue, it would be nice to have a solution in the interim. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5848) MapReduce counts forcibly preempted containers as FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036692#comment-14036692 ] Ashwin Shankar commented on MAPREDUCE-5848: --- Committers, can we please commit this if its looking good ? I tried out this patch and it seems to work well. Without this,jobs fail due to preemption after mapreduce.(map,reduce).maxattempts limit is reached. This is also blocking YARN-1961. MapReduce counts forcibly preempted containers as FAILED Key: MAPREDUCE-5848 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5848 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Carlo Curino Assignee: Subramaniam Venkatraman Krishnan Attachments: MR-5848.patch, MR-5848.patch, YARN-1958.patch The MapReduce AM is considering a forcibly preempted container as FAILED, while I think it should be considered as KILLED (i.e., not count against the maximum number of failures). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-7-b23.txt Thanks a lot Jason. I've attached the patch for 23. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt, MAPREDUCE-5251-7-b23.txt, MAPREDUCE-5251-7.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5411: -- Attachment: LOADED_JOB_CACHE_MR5411-2.txt Refresh size of loaded job cache on history server -- Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: LOADED_JOB_CACHE_MR5411-1.txt, LOADED_JOB_CACHE_MR5411-2.txt We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5425) Junit in TestJobHistoryServer failing in jdk 7
Ashwin Shankar created MAPREDUCE-5425: - Summary: Junit in TestJobHistoryServer failing in jdk 7 Key: MAPREDUCE-5425 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5425 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.0.4-alpha Reporter: Ashwin Shankar We get the following exception when we run the unit tests of TestJobHistoryServer with jdk 7: Caused by: java.net.BindException: Problem binding to [0.0.0.0:10033] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:719) at org.apache.hadoop.ipc.Server.bind(Server.java:423) at org.apache.hadoop.ipc.Server$Listener.init(Server.java:535) at org.apache.hadoop.ipc.Server.init(Server.java:2202) at org.apache.hadoop.ipc.RPC$Server.init(RPC.java:901) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.init(ProtobufRpcEngine.java:505) at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:480) at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:746) at org.apache.hadoop.mapreduce.v2.hs.server.HSAdminServer.serviceInit(HSAdminServer.java:100) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) This is happening because testMainMethod starts the history server and doesnt stop it. This worked in jdk 6 because tests executed sequentially and this test was last one and didnt affect other tests,but in jdk 7 it fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5411: -- Status: Patch Available (was: Open) Thanks,patch refreshed.. Refresh size of loaded job cache on history server -- Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: LOADED_JOB_CACHE_MR5411-1.txt, LOADED_JOB_CACHE_MR5411-2.txt We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION--5.txt Thanks,patch updated. Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt, JOB_RETENTION-3.txt, JOB_RETENTION-4.txt, JOB_RETENTION--5.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-7.txt Thanks,patch refreshed. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt, MAPREDUCE-5251-7.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION-4.txt Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt, JOB_RETENTION-3.txt, JOB_RETENTION-4.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
Ashwin Shankar created MAPREDUCE-5411: - Summary: Refresh size of loaded job cache on history server Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5411: -- Status: Patch Available (was: Open) Added a new command on history server's admin interface 'refreshLoadedJobCache' which refreshes the size of loaded job cache. Refresh size of loaded job cache on history server -- Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: LOADED_JOB_CACHE_MR5411-1.txt We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5411: -- Attachment: LOADED_JOB_CACHE_MR5411-1.txt Refresh size of loaded job cache on history server -- Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: LOADED_JOB_CACHE_MR5411-1.txt We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5411: -- Attachment: LOADED_JOB_CACHE_MR5411-1.txt Refresh size of loaded job cache on history server -- Key: MAPREDUCE-5411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: LOADED_JOB_CACHE_MR5411-1.txt We want to be able to refresh size of the loaded job cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-5.txt Thanks,patch updated. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-6.txt Makes sense,both the comments addressed in latest patch. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION-3.txt Fixed the javadoc warning. Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt, JOB_RETENTION-3.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION-2.txt Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Attachment: WHOLE_PATCH_NOT_TO_BE_CHKEDIN-MAPREDUCE-5356-5.txt Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt, MAPREDUCE-5266-4.txt, MAPREDUCE-5356-5.txt, WHOLE_PATCH_NOT_TO_BE_CHKEDIN-MAPREDUCE-5356-5.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712540#comment-13712540 ] Ashwin Shankar commented on MAPREDUCE-5356: --- Thanks Jason,all your comments are addressed in the latest patch. Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt, MAPREDUCE-5266-4.txt, MAPREDUCE-5356-5.txt, WHOLE_PATCH_NOT_TO_BE_CHKEDIN-MAPREDUCE-5356-5.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Attachment: MAPREDUCE-5356-5.txt Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt, MAPREDUCE-5266-4.txt, MAPREDUCE-5356-5.txt, WHOLE_PATCH_NOT_TO_BE_CHKEDIN-MAPREDUCE-5356-5.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-13.txt Marked new class as @Private. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-10.txt, JHS_REFRESH-12.txt, JHS_REFRESH-13.txt, JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-4.txt Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712772#comment-13712772 ] Ashwin Shankar commented on MAPREDUCE-5251: --- Thanks,patch updated ! Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-12.txt Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-10.txt, JHS_REFRESH-12.txt, JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a usernam
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711795#comment-13711795 ] Ashwin Shankar commented on MAPREDUCE-5265: --- Thanks. I've modified the patch to use the protobuf code which got moved to hadoop-common for GetUserMappings and RefreshUserMappings. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-10.txt, JHS_REFRESH-12.txt, JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Attachment: MAPREDUCE-5266-4.txt Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt, MAPREDUCE-5266-4.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-10.txt Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-10.txt, JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a usernam
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707183#comment-13707183 ] Ashwin Shankar commented on MAPREDUCE-5265: --- Thanks, I've incorporated all your comments except one mentioned below. bq. RefreshUserMappingsProtocol.proto, GetUserMappingsPrototcol.proto, and the supporting glue code is duplicated from HDFS – is there a way we can simply use the HDFS version? I see the PB glue in HDFS is already marked LimitedPrivate for HDFS and MapReduce, so it seems like we should just be using that rather than duplicating it if possible. I actually tried this when I started working on this JIRA,however I found that I was not able to use HDFS classes in History server since the former is not added as a dependency to the latter,which makes sense. Ideally we would want move these protocol classes to common and then derive it in HDFS and History server. I'm planning to file a follow-up JIRA to do that. But for now,I felt that its best to duplicate this part of the code. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-10.txt, JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
Ashwin Shankar created MAPREDUCE-5386: - Summary: Refresh job retention time,job cleaner interval, enable/disable cleaner Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Reporter: Ashwin Shankar Assignee: Ashwin Shankar We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION-1.txt Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Affects Version/s: 2.1.0-beta Status: Patch Available (was: Open) This patch will not apply now since it depends on MAPREDUCE-5265. I've created a new option on the history server admin interface : '-refreshJobRetentionSettings' which will refresh all the three parameters mentioned in the description. I've added two unit tests - one tests the job history retention stuff in isolation and the other tests the RPC communication. Refresh job retention time,job cleaner interval, enable/disable cleaner --- Key: MAPREDUCE-5386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: JOB_RETENTION-1.txt We want to be able to refresh following job retention parameters without having to bounce the history server : 1. Job retention time - mapreduce.jobhistory.max-age-ms 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695550#comment-13695550 ] Ashwin Shankar commented on MAPREDUCE-5356: --- Thanks for your comment Devaraj. So we then have two options here : 1. We don't include LOG_AGGREGATION_ENABLED property as part of the refresh. Currently in trunk if you look at AggregatedLogDeletionService#serviceStart(),we schedule the logDeletionTask only if LOG_AGGREGATION_ENABLED(LAE) is set to true. Then I would need to the same check at the start of my refresh code ie dont refresh 'log retention','check interval' prop AND dont start the deletion task if LAE property is false. But this means - '-refreshLogRetentionSettings' depends on LAE and would be useless until we start history server with LAE set to true, AND we need to bounce the history server and NM if we want to change LAE. 2. Second option, is to make two rpc calls -one to HS and one to NM to do the refresh on LAE. But as far as I looked, I don't see an admin interface on NM to do that. We could probably create another JIRA to create an RPC server on NM and have a separate command something like '-refreshLogAggregationEnable' . What are your thoughts ? I'll post a new patch based on my point-1 if thats agreeable. Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
Ashwin Shankar created MAPREDUCE-5356: - Summary: Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Status: Patch Available (was: Open) Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Attachment: MAPREDUCE-5266-2.txt Note: This patch depends on MAPREDUCE-5265 and will not apply now. HadoopQA will complain. Summary:I have added a new command '-refreshLogSettings' to the history server admin interface which would refresh both: 1.log retention period(yarn.log-aggregation.retain-seconds) 2. check interval(yarn.log-aggregation.retain-check-interval-seconds) I created a single command for both the properties because : 1. Both properties are related. More often than not,if we change one of them we would change the other. 2.it seemed better from a usability perspective since the user needs to execute just one command for both the log properties instead of two separate commands. I've added two tests - one tests AggregatedLogDeletionService in isolation and validates if the refreshLogSettings is working properly. The other test validates the rpc communication between the job history client and server for this new protocol. Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-3.txt Makes sense,thanks for your comments. I've made the suggested changes. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5356) Refresh Log aggregation 'retention period' and 'check interval'
[ https://issues.apache.org/jira/browse/MAPREDUCE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5356: -- Attachment: MAPREDUCE-5266-3.txt I just figured that it made sense to add the LOG_AGGREGATION_ENABLED property(yarn.log-aggregation-enable) to be refreshed along with other two props mentioned in my first comment. So,I've added that. Also, there was room for refactoring in AggregatedLogDeletionService.java,I've done that as well. Refresh Log aggregation 'retention period' and 'check interval' Key: MAPREDUCE-5356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5356 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Ashwin Shankar Assignee: Ashwin Shankar Labels: features Attachments: MAPREDUCE-5266-2.txt, MAPREDUCE-5266-3.txt We want to be able to refresh log aggregation retention time and 'check interval' time on the fly by changing configs so that we dont have to bounce history server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-9.txt Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt, JHS_REFRESH-9.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-8.txt Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Target Version/s: 2.1.0-beta, 0.23.9 (was: 0.23.8, 2.0.5-alpha) Status: Patch Available (was: Open) I have changed my rpc layer implementation to use classes from hadoop-common rather than Yarn rpc. I've also incorporated the comments suggested. This patch includes units tests. However please note that this patch doesn't apply to branch-23. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a usernam
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692488#comment-13692488 ] Ashwin Shankar commented on MAPREDUCE-5265: --- All the findbugs warnings are due to sources generated by ProtocolBuffers. How do we handle this ? Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt, JHS_REFRESH-8.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a usernam
[
https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686974#comment-13686974
]
Ashwin Shankar commented on MAPREDUCE-5265:
---
{quote}
protocol files are in package org.apache.hadoop.yarn but
org.apache.hadoop.mapreduce is more appropriate. The history server is specific
to MapReduce and not a generic YARN history server. In the future YARN may
provide its own, separate history server
{quote}
YARN rpc has hardcoded the protocol package name to
org.apache.hadoop.yarn(RpcServerFactoryPBImpl.java). So things wont work if I
change history server's protocol package name to *mapreduce.
Need an admin interface on history server with the ability to refresh super
user groups,refresh user to group mappings,refresh admin acls,get groups
given a username.
--
Key: MAPREDUCE-5265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobhistoryserver
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Assignee: Ashwin Shankar
Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt
The history server needs an admin interface with the ability to
1. refresh the super user groups configurations,
2. refresh user to group mappings,
3. refresh its admin acls,
4. get groups given a username
without requiring a restart of the history server. This is analogous to the
-refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin
and yarn rmadmin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Attachment: MAPREDUCE-5316-1-b23.txt
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
Also,we could make input task-state to be case-insensitive to be consistent
with
task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Target Version/s: 2.1.0-beta, 0.23.9
Status: Patch Available (was: Open)
I've added code to validate task-state and print error/usage messages for
invalid input.
There is a small catch here- branch-23 didn't have any validation code for
checking task-states,while trunk had one. However,in trunk, the validation code
threw an exception. I've modified trunk as well to print error/usage message
rather than throw an exception,so that we have consistent behavior across
branches. Also,I've made task-state case-insensitive.
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.8, 2.0.4-alpha, 3.0.0
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
Also,we could make input task-state to be case-insensitive to be consistent
with
task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-2.txt Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Target Version/s: 0.23.8, 2.0.4-alpha Status: Patch Available (was: Open) The issue here is was that any IOException that arises from Fetcher.copyMapOutput is always blamed on map causing the map to be re-run. But we can get an exception like DiskErrorException which is due to lack of disk space on the reduce node and has nothing to do with map. The map task should not be re-executed in this case. To solve this,I've added code to catch this DiskErrorException,blame the reduce and report this to the shuffle scheduler ,which in turn kills this reduce task and would hopefully re-run it on another node which has disk space. I manually tested this by changing the source and stubbing it to reserve 1 Peta-byte during the shuffle phase. This throws the DiskErrorException as expected,blames the reduce and most importantly doesn't run the map task again. I've also added a unit test to simulate this scenario. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.4-alpha, 0.23.7 Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682631#comment-13682631 ] Ashwin Shankar commented on MAPREDUCE-5251: --- Also please note that this patch doesn't directly apply to branch-23. I'll upload a separate patch once all the comments are resolved for this current patch. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-4019:
--
Attachment: MAPREDUCE-4019-b23-2.txt
MAPREDUCE-4019-b23-2.txt is the patch for branch-23. Please note that Jenkins
will complain since this won't apply to trunk.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
Ashwin Shankar created MAPREDUCE-5316:
-
Summary: job -list-attempt-ids command does not handle illegal
task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.8, 2.0.4-alpha, 3.0.0
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Description:
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
was:
Courtesy : [~mikanboy]]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Description:
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
was:
Courtesy : [~mikanboy]]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Affects Version/s: (was: 2.0.4-alpha)
(was: 3.0.0)
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Affects Version/s: 3.0.0
2.0.4-alpha
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Affects Version/s: 3.0.0
2.0.4-alpha
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Affects Version/s: (was: 2.0.4-alpha)
(was: 3.0.0)
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-5316:
--
Description:
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
Also,we could make input task-state to be case-insensitive to be consistent with
task-type behavior(MAPREDUCE-4019).
was:
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Exception in thread main java.lang.IllegalArgumentException: No enum const
class org.apache.hadoop.mapreduce.TaskType.map
at java.lang.Enum.valueOf(Enum.java:196)
at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
1) illegal task-state not handled.
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 MAP complete
12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
{quote}
job -list-attempt-ids command does not handle illegal task-state
Key: MAPREDUCE-5316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ashwin Shankar
Assignee: Ashwin Shankar
Courtesy : [~mikanboy]
{quote}
job -list-attempt-ids command should handle illegal argument for task-state
the same way as task-type. Right now only illegal task-type is handle by
an exception being thrown. Illegal task-state on the other hand does not
throw exception. For example is a user mistype 'completed' as 'complete', they
may wrongly think there are no completed tasks, instead of being notified of
the illegal task-state that was used.
1) illegal task-type handled.
[philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
/home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
job_1345673924741_0086 map completed
12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead,
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-6.txt Patch updated.I was reusing a class 'ProtoBase' which unfortunately got moved to different package in the last few hours :) causing my previous patch to fail. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt, JHS_REFRESH-6.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar reassigned MAPREDUCE-4019:
-
Assignee: Ashwin Shankar
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019.patch,
MAPREDUCE-4019.patch, MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-4019:
--
Attachment: MAPREDUCE-4019-6.txt
Thanks Jason, patch updated. I forgot to mention this before,this patch doesn't
apply to branch-23 . I'll upload a separate patch for 23 once I gets the green
signal for this one.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019.patch, MAPREDUCE-4019.patch, MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-4.txt Patch updated.My bad,I was testing the patch with older version of trunk. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt, JHS_REFRESH-4.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5265) Ability to refresh super user groups on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar reassigned MAPREDUCE-5265: - Assignee: Ashwin Shankar Ability to refresh super user groups on history server -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar The history server needs the ability to refresh the super user groups configurations without requiring a restarting the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5266) Ability to refresh retention settings on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5266: -- Assignee: Ashwin Shankar Ability to refresh retention settings on history server --- Key: MAPREDUCE-5266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5266 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar It would be very useful if the job and log retention settings of the history server could be refreshed without restarting the history server. This would include such things as: * how many to jobs to keep for browsing * how many jobs to cache * how long to retain jobs * how long to retain logs * how often to check for retention -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Description: The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restarting the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. was:The history server needs the ability to refresh the super user groups configurations without requiring a restarting the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. Summary: Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. (was: Ability to refresh super user groups on history server) I've changed the summary and description to reflect all the features which this patch is going to contain. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restarting the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Description: The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. was: The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restarting the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Attachment: JHS_REFRESH-2.txt This patch(JHS_REFRESH-2.txt) contains all the features mentioned in the description. On a high level I've changed 'mapred' shell script to incorporate the new 'hsadmin' commands. I've created a client(which is instantiated by shell script),the client rpc stub,protocol buffer rpc messages,server rpc stub,admin server which executes the command.NOTE : I've not written unit tests yet. I wanted to get comments first so that I don't have to rewrite tests in case there are major comments . Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5265) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5265: -- Target Version/s: 0.23.8, 2.0.5-alpha Status: Patch Available (was: Open) Need an admin interface on history server with the ability to refresh super user groups,refresh user to group mappings,refresh admin acls,get groups given a username. -- Key: MAPREDUCE-5265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5265 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: JHS_REFRESH-2.txt The history server needs an admin interface with the ability to 1. refresh the super user groups configurations, 2. refresh user to group mappings, 3. refresh its admin acls, 4. get groups given a username without requiring a restart of the history server. This is analogous to the -refreshSuperUserGroupsConfiguration capabilities provided by hdfs dfsadmin and yarn rmadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-4019:
--
Attachment: MAPREDUCE-4019.txt
Hi,
I wanted to help move this patch. I've made changes to remove
JOB_SETUP,JOB_CLEANUP options. I've added code to validate task-type options at
the beginning(CLI.run()) rather than passing it to other classes which would
throw an exception somewhere down the call hierarchy(current implementation).
I've also made the options for --list-attempt-ids case insensitive, so options
'map','reduce' will work. I've added couple of unit tests to test all these
changes.Please note that I've reused some of the code from the previous patch
contributed by B.Kumar.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Priority: Minor
Attachments: MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-4019:
--
Attachment: MAPREDUCE-4019-4.txt
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019.patch,
MAPREDUCE-4019.patch, MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar reassigned MAPREDUCE-5251: - Assignee: Ashwin Shankar Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4927) Historyserver 500 error due to NPE when accessing specific counters page for failed job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-4927: -- Attachment: MAPREDUCE-4927.txt The problem is that a failed task doesn't have counters and we assume that we always get counters which causes an NPE. I've added a null check for counters to fix this. Also I've changed a unit test to incorporate this case. Historyserver 500 error due to NPE when accessing specific counters page for failed job --- Key: MAPREDUCE-4927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4927 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.0.3-alpha, 0.23.6 Reporter: Jason Lowe Attachments: MAPREDUCE-4927.txt Went to the historyserver page for a job that failed and examined the counters page. When I clicked on a specific counter, the historyserver returned a 500 error. The historyserver logs showed it encountered an NPE error, full traceback to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4927) Historyserver 500 error due to NPE when accessing specific counters page for failed job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-4927: -- Assignee: Ashwin Shankar Target Version/s: 2.0.5-beta, 0.23.8 Status: Patch Available (was: Open) Historyserver 500 error due to NPE when accessing specific counters page for failed job --- Key: MAPREDUCE-4927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4927 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.6, 2.0.3-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-4927.txt Went to the historyserver page for a job that failed and examined the counters page. When I clicked on a specific counter, the historyserver returned a 500 error. The historyserver logs showed it encountered an NPE error, full traceback to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
