[jira] [Created] (MAPREDUCE-6650) Surface error histograms from the AM

Bikas Saha created MAPREDUCE-6650: - Summary: Surface error histograms from the AM Key: MAPREDUCE-6650 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6650 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-3755) Add the equivalent of JobStatus to end of JobHistory file

[ https://issues.apache.org/jira/browse/MAPREDUCE-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-3755. --- Resolution: Won't Fix Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0

RE: [VOTE] Release Apache Hadoop 2.4.0

+1 (binding). Ran it with Apache Tez. I agree with Suresh about including only critical regressions/bugs at this point. The less code we change the less new bugs we bring in at the last moment. Bikas -Original Message- From: Jonathan Eagles [mailto:jeag...@gmail.com] Sent: Monday, April

RE: About block splitting, input split and TextInputFormat in MapReduce

The overall answer is that InputFormat implementations determine how to split their data across block boundaries and then handle the read in order to not have incomplete records. When splits are generated then they typically don’t have block information. They have an offset and length into the f

RE: [VOTE] Release Apache Hadoop 2.2.0

+1 (non binding) -Original Message- From: Arpit Gupta [mailto:ar...@hortonworks.com] Sent: Thursday, October 10, 2013 10:06 AM To: common-...@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Had

RE: [VOTE] Release Apache Hadoop 2.1.0-beta

+1 (non-binding) Bikas -Original Message- From: Hitesh Shah [mailto:hit...@apache.org] Sent: Wednesday, August 21, 2013 4:25 PM To: yarn-...@hadoop.apache.org; Arun Murthy Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org Subject: Re: [VOTE] Rel

RE: Abstraction layer to support both YARN and Mesos

+1 for Tom's suggestion. That is how we have transparently redirected MR jobs to use Tez as the execution framework. Bikas -Original Message- From: Tom White [mailto:t...@cloudera.com] Sent: Wednesday, July 31, 2013 8:41 AM To: mapreduce-dev Cc: yarn-...@hadoop.apache.org Subject: Re: Abs

RE: [VOTE] Release Apache Hadoop 2.1.0-beta

+1. Using it locally for Tez. Bikas -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Tuesday, July 30, 2013 6:30 AM To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org Subject: [VOTE] Release

[jira] [Resolved] (MAPREDUCE-5398) MR changes for YARN-513

[ https://issues.apache.org/jira/browse/MAPREDUCE-5398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-5398. --- Resolution: Fixed Fix Version/s: 2.1.0-beta Target Version/s: 2.1.0-beta

[jira] [Created] (MAPREDUCE-5398) MR changes for YARN-513

Bikas Saha created MAPREDUCE-5398: - Summary: MR changes for YARN-513 Key: MAPREDUCE-5398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5398 Project: Hadoop Map/Reduce Issue Type: Bug

[jira] [Created] (MAPREDUCE-5314) Fix build break by YARN-686 (flatten node report)

Bikas Saha created MAPREDUCE-5314: - Summary: Fix build break by YARN-686 (flatten node report) Key: MAPREDUCE-5314 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5314 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-5161) Merge MAPREDUCE-1806 from branch-1 to branch-1-win. CombineFileInputFormat fix for paths not on default FS

[ https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-5161. --- Resolution: Fixed Fix Version/s: 1-win +1. Committed to branch-1-win

[jira] [Resolved] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks

[ https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-4987. --- Resolution: Fixed Fix Version/s: 3.0.0 +1. Committed to trunk

[jira] [Resolved] (MAPREDUCE-5140) MR part of YARN-514

[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-5140. --- Resolution: Fixed Committed to trunk and branch-2 > MR part of YARN-

[jira] [Created] (MAPREDUCE-5115) MR app master may try to assign a reduce priority container to a map

Bikas Saha created MAPREDUCE-5115: - Summary: MR app master may try to assign a reduce priority container to a map Key: MAPREDUCE-5115 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5115 Project

[jira] [Created] (MAPREDUCE-5107) MR AM needs to be able to handle unavailable HDFS

Bikas Saha created MAPREDUCE-5107: - Summary: MR AM needs to be able to handle unavailable HDFS Key: MAPREDUCE-5107 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5107 Project: Hadoop Map/Reduce

RE: [Vote] Merge branch-trunk-win to trunk

ck, including scale tests. This is made possible only with the contribution from many many folks in the community. Following people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur Reddy Bolli,

[jira] [Created] (MAPREDUCE-4893) MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality

Bikas Saha created MAPREDUCE-4893: - Summary: MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality Key: MAPREDUCE-4893 URL: https://issues.apache.org/jira/browse

[jira] [Created] (MAPREDUCE-4892) CombineFileInputFormat node input split can be skewed on small clusters

Bikas Saha created MAPREDUCE-4892: - Summary: CombineFileInputFormat node input split can be skewed on small clusters Key: MAPREDUCE-4892 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4892

RE: How to compile the JobTracker.java

The scheduling and resource management parts of the JobTracker have been put in the ResourceManager. The Map Reduce job history in the JobHistoryServer. Each MR job is managed by its own YARN MR Application. Bikas -Original Message- From: Tiago Kepe [mailto:tiagok...@gmail.com] Sent: Tues

[jira] [Created] (MAPREDUCE-4671) AM does not tell the RM about container requests that are no longer needed

Bikas Saha created MAPREDUCE-4671: - Summary: AM does not tell the RM about container requests that are no longer needed Key: MAPREDUCE-4671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671

RE: More than one RM in YARN?

em to be implemented, I decided to just use DNS and a manual failover. Simple is best, I suppose. Cheers, Chris On 9/19/12 10:18 AM, "Bikas Saha" wrote: >If I understand you correctly, your target scenario is failover? > >Its good to know that you have something

RE: More than one RM in YARN?

/19/12 10:05 AM, "Bikas Saha" wrote: >Chris, > >Could you please elaborate a bit on the use case you are targeting? Do >you want to run multiple active RM's in the cluster? And they are being >used concurrently? Or are you talking about failover scenarios? > >

RE: More than one RM in YARN?

Chris, Could you please elaborate a bit on the use case you are targeting? Do you want to run multiple active RM's in the cluster? And they are being used concurrently? Or are you talking about failover scenarios? Bikas -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: T

[jira] [Created] (MAPREDUCE-4657) WindowsResourceCalculatorPlugin has NPE

Bikas Saha created MAPREDUCE-4657: - Summary: WindowsResourceCalculatorPlugin has NPE Key: MAPREDUCE-4657 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4657 Project: Hadoop Map/Reduce

[jira] [Resolved] (MAPREDUCE-4561) Support for node health scripts on Windows

[ https://issues.apache.org/jira/browse/MAPREDUCE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-4561. --- Resolution: Duplicate > Support for node health scripts on Wind

[jira] [Created] (MAPREDUCE-4635) MR side of YARN-83. Changing package of YarnClient

Bikas Saha created MAPREDUCE-4635: - Summary: MR side of YARN-83. Changing package of YarnClient Key: MAPREDUCE-4635 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4635 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-4598) Support for node health scripts on Windows

Bikas Saha created MAPREDUCE-4598: - Summary: Support for node health scripts on Windows Key: MAPREDUCE-4598 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4598 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-4597) TestKillSubProcesses intermittently fails

Bikas Saha created MAPREDUCE-4597: - Summary: TestKillSubProcesses intermittently fails Key: MAPREDUCE-4597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4597 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-4564) Shell timeout mechanism does not work for processes spawned using winutils

Bikas Saha created MAPREDUCE-4564: - Summary: Shell timeout mechanism does not work for processes spawned using winutils Key: MAPREDUCE-4564 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4564

[jira] [Created] (MAPREDUCE-4561) Support for node health scripts on Windows

Bikas Saha created MAPREDUCE-4561: - Summary: Support for node health scripts on Windows Key: MAPREDUCE-4561 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4561 Project: Hadoop Map/Reduce

RE: Multi-level aggregation with combining the result of maps per node/rack

Can you please share a brief note on the design. Just a few sentences on the main changes. What you are saying sounds similar to multi-level aggregation done in the Dryad runtime. That is useful to reduce the input arity (as you suggest) and al

[jira] [Created] (MAPREDUCE-4472) Support secure AM launch for unmanaged AM's

Bikas Saha created MAPREDUCE-4472: - Summary: Support secure AM launch for unmanaged AM's Key: MAPREDUCE-4472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4472 Project: Hadoop Map/R

[jira] [Created] (MAPREDUCE-4438) Add client side for UnmanagedRM

Bikas Saha created MAPREDUCE-4438: - Summary: Add client side for UnmanagedRM Key: MAPREDUCE-4438 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4438 Project: Hadoop Map/Reduce Issue

[jira] [Created] (MAPREDUCE-4436) AppRejectedTransition does not unregister app from master service and scheduler

Bikas Saha created MAPREDUCE-4436: - Summary: AppRejectedTransition does not unregister app from master service and scheduler Key: MAPREDUCE-4436 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4436

[jira] [Created] (MAPREDUCE-4427) Enable the RM to work with AM's that are not managed by it

Bikas Saha created MAPREDUCE-4427: - Summary: Enable the RM to work with AM's that are not managed by it Key: MAPREDUCE-4427 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427 Project: H

[jira] [Created] (MAPREDUCE-4401) Enhancements to HDFS for Windows Server and Windows Azure development and runtime environments

Bikas Saha created MAPREDUCE-4401: - Summary: Enhancements to HDFS for Windows Server and Windows Azure development and runtime environments Key: MAPREDUCE-4401 URL: https://issues.apache.org/jira/browse/MAPREDUCE

Re: Hadoop 2.0 / YARN - getting full node report in application master

in that >regard. > >A reasonable assumption could be that the first time you call it that you >get a full snapshot. > >Presumably the second (and later) calls return updates since the previous >call, but even that isn't clear. > >On 6/22/12 3:29 PM, "Bikas Saha"

Re: Hadoop 2.0 / YARN - getting full node report in application master

AMResponse.getUpdatedNodes() would give you only nodes that have been updated. Say they are lost or become healthy again. So you won't get total cluster information from that. On 6/22/12 3:10 PM, "Thomas Weise" wrote: >Hello, > >I'm looking for a way to initially obtain a full report of cluste

[jira] [Resolved] (MAPREDUCE-2713) Recovery of ResourceManager

[ https://issues.apache.org/jira/browse/MAPREDUCE-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-2713. --- Resolution: Duplicate Resolving as dup of MAPREDUCE-4326 > Recovery

Re: single node cluster - windows

Hi, If you build your own code then you can sync to branch-1-win on Hadoop and build it. Hadoop is being actively ported to run natively on Windows on that branch. You will need Windows platform SDK on the path. Visual Studio Express (free) will work. To build ant -dwindows=true jar Bikas On 6

[jira] [Resolved] (MAPREDUCE-4263) Use taskkill /T to terminate tasks on Windows

[ https://issues.apache.org/jira/browse/MAPREDUCE-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-4263. --- Resolution: Fixed Fixed in the change for MAPREDUCE-4260 > Use taskk

[jira] [Created] (MAPREDUCE-4331) TaskAttemptListenerImpl.statusUpdate and TaskAttemptListenerImpl.ping never return false

Bikas Saha created MAPREDUCE-4331: - Summary: TaskAttemptListenerImpl.statusUpdate and TaskAttemptListenerImpl.ping never return false Key: MAPREDUCE-4331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4331

[jira] [Created] (MAPREDUCE-4330) TaskAttemptCompletedEventTransition invalidates previously successful attempt without checking if the newly completed attempt is successful

Bikas Saha created MAPREDUCE-4330: - Summary: TaskAttemptCompletedEventTransition invalidates previously successful attempt without checking if the newly completed attempt is successful Key: MAPREDUCE-4330 URL

[jira] [Created] (MAPREDUCE-4263) Use taskkill /T to terminate tasks on Windows

Bikas Saha created MAPREDUCE-4263: - Summary: Use taskkill /T to terminate tasks on Windows Key: MAPREDUCE-4263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4263 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-4259) TestDatanodeBlockScanner and TestReplication fail intermittently on Windows

Bikas Saha created MAPREDUCE-4259: - Summary: TestDatanodeBlockScanner and TestReplication fail intermittently on Windows Key: MAPREDUCE-4259 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4259

Re: OS Independent container monitor

https://issues.apache.org/jira/browse/MAPREDUCE-4204 tracks something on these lines for the port to Windows. On 4/25/12 2:49 PM, "Radim Kolar" wrote: >Currently contained monitor works only on Linux. > >Looking at code >org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Contai

[jira] [Created] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable

Bikas Saha created MAPREDUCE-4204: - Summary: Refactor ProcfsBasedProcessTree to make the resource collection object pluggable Key: MAPREDUCE-4204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204

[jira] [Created] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows

Bikas Saha created MAPREDUCE-4203: - Summary: Create equivalent of ProcfsBasedProcessTree for Windows Key: MAPREDUCE-4203 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 Project: Hadoop Map

[jira] [Created] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working

Bikas Saha created MAPREDUCE-4201: - Summary: Getting PID not working on Windows. Termination of Task/TaskJVM's not working Key: MAPREDUCE-4201 URL: https://issues.apache.org/jira/browse/MAPREDUCE

Re: OS Independent container monitor

+1 for making ProcfsBasedProcessTree an implementation of a pluggable interface. Will help projects like the port to Windows. On 4/25/12 11:14 PM, "Harsh J" wrote: >I like the idea, and more portable code is good (although we should >also allow per-platform overrides in case that can provide bet

[jira] [Created] (MAPREDUCE-4128) AM Recovery expects all attempts of a completed task to also be completed.

/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 3.0.0 The AM seems to assume that all attempts of a completed task (from a previous AM incarnation) would also be completed

[jira] [Created] (MAPREDUCE-3986) TaskAttemptCompletionEvent not being sent for task attempts that were killed after a sibling attempt succeeds

://issues.apache.org/jira/browse/MAPREDUCE-3986 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 0.23.2 When TaskImpl transitions to SUCCEEDED then it kills the other running

[jira] [Resolved] (MAPREDUCE-3985) handleTaskAttemptCompletion() called twice for same task attempt from KillWaitAttemptKilledTransition & AttemptKilledTransition

[ https://issues.apache.org/jira/browse/MAPREDUCE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-3985. --- Resolution: Not A Problem Not an issue. I got confused between AttemptKilledTransition

[jira] [Created] (MAPREDUCE-3985) handleTaskAttemptCompletion() called twice for same task attempt from KillWaitAttemptKilledTransition & AttemptKilledTransition

3985 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3985 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 0.23.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 0.23.2 Task

[jira] [Created] (MAPREDUCE-3976) TestRMContainerAllocator failing

: 0.23.0 Reporter: Bikas Saha Assignee: Jason Lowe Fix For: 0.23.0 The following stack trace is being generated org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics already exists! at org.apache.hadoop.metrics2

[jira] [Created] (MAPREDUCE-3958) Remove RMNodeState and replace it with NodeState

: mrv2 Affects Versions: 0.23.0 Reporter: Bikas Saha Fix For: 0.23.2 RMNodeState is being sent over the wire after MAPREDUCE-3353. This has been done by cloning the enum into NodeState in yarn protocol records. That makes RMNodeState redundant and it should be replaced

[jira] [Created] (MAPREDUCE-3923) RM/AM/JHS webserver

RM/AM/JHS webserver --- Key: MAPREDUCE-3923 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3923 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha -- This message is automatically generated

[jira] [Created] (MAPREDUCE-3899) Locking not correct in org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(AllocateRequest request)

URL: https://issues.apache.org/jira/browse/MAPREDUCE-3899 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For