[jira] [Resolved] (MAPREDUCE-3936) Clients should not enforce counter limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3936. -- Resolution: Won't Fix Clients should not enforce counter limits -- Key: MAPREDUCE-3936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Reporter: Tom White Assignee: Tom White Labels: BB2015-05-TBR Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf instance to load the limits, which may throw an exception if the client limit is set to be lower than the limit on the cluster (perhaps because the cluster limit was raised from the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Hadoop 2.0.4-alpha
+1 Checked sigs and checksums, source tag, and built from source. Cheers, Tom On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like to release. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
[jira] [Created] (MAPREDUCE-5035) Update MR1 memory configuration docs
Tom White created MAPREDUCE-5035: Summary: Update MR1 memory configuration docs Key: MAPREDUCE-5035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Tom White Assignee: Tom White The pmem/vmem settings in the docs (http://hadoop.apache.org/docs/r1.1.1/cluster_setup.html#Memory+monitoring) have not been supported for a long time. The docs should be updated to reflect the new settings (mapred.cluster.map.memory.mb etc). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5035) Update MR1 memory configuration docs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-5035. -- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I just committed this. Update MR1 memory configuration docs Key: MAPREDUCE-5035 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5035 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Tom White Assignee: Tom White Fix For: 1.2.0 Attachments: cluster_setup.html, MAPREDUCE-5035.patch The pmem/vmem settings in the docs (http://hadoop.apache.org/docs/r1.1.1/cluster_setup.html#Memory+monitoring) have not been supported for a long time. The docs should be updated to reflect the new settings (mapred.cluster.map.memory.mb etc). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Release numbering for branch-2 releases
Possibly the reason for Stack's consternation is that this is a Hadoop-specific versioning scheme, rather than a standard one like Semantic Versioning (http://semver.org/) which is more widely understood. With that scheme we would have something like 2.0.0-alpha, 2.0.0-alpha.1, 2.0.0-alpha.2, 2.0.0-alpha.3, 2.0.0-beta, 2.0.0 so that the alpha and beta tags all precede the 2.0.0 GA release, which is the one that we make compatibility promises for. Whereas Arun is proposing 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.1.0-alpha, 2.2.0-beta, 2.3.0 and the casual observer might expect there to be a stable 2.0.1 (say) on seeing the existence of 2.0.2-alpha. The first three of these are already released, so I don't think we could switch to the Semantic Versioning scheme at this stage. We could for release 3 though. Tom On Thu, Jan 31, 2013 at 8:12 PM, Arun C Murthy a...@hortonworks.com wrote: Stack, On Jan 30, 2013, at 9:25 PM, Stack wrote: I find the above opaque and written in a cryptic language that I might grok if I spent a day or two running over cited issues trying to make some distillation of the esotericia debated therein. If you want feedback from other than the cognescenti, I would suggest a better summation of what all is involved. I apologize if there was too much technical details. The simplified version is that hadoop-2 isn't baked as it stands today, and is not viable to be supported by this community in a stable manner. In particular, it is due to the move to PB for HDFS protocols and the freshly minted YARN apis/protocols. As a result, we have been forced to make (incompatible) changes in every hadoop-2 release so far (2.0.0, 2.0.2 etc.). Since we released the previous bits we have found security issues, bugs and other issues which will cause long-term maintenance harm (details are in the HADOOP/HDFS/YARN jiras in the original email). My aim, as the RM, is to try nudge (nay, force) all contributors to spend time over the next couple of months focussing on fixing known issues and to look for other surprises - this way I hope to ensure we do not have further incompatible changes for downstream projects and we can support hadoop-2 for at least a couple of years. I hope this makes sense to you. I don't think turning around and calling these 3.x or 4.x makes things better since no amount of numbering lipstick will make the software better or viable for the long-term for both users and other projects. Worse, it will force HBase and other projects to deal with *even more* major Hadoop releases... which seems like a royal pita. I hope that clarifies things. Thanks Stack. Arun
Re: Release numbering for branch-2 releases
On Wed, Jan 30, 2013 at 11:32 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: I still have a list of pending API/protocol cleanup in YARN that need to be in before we even attempt supporting compatibility further down the road. To let others track these it would be useful if they were tagged in JIRA with a label (e.g. apichange). There's no way we can support wire compatibility with the APIs in the state that they are in now. So, +1 for a beta sometime in March. There are some early adopters, I am particularly speaking of YARN, who have been instrumental in helping ironing out the alpha software, some with very large clusters and end-user base. These users will continue to be affected with these API/protocol changes, but the alpha tag was clearly meant to clarify this. I think we should graciously send out a note (on general@) about an impending beta from where everyone can except a high degree of compatibility. Just caught up with the discussion on the referred JIRAs. I can clearly see how a single release with an umbrella alpha/beta tag is causing tensions *only* because we have a single project and product. More reinforcement for my proclivity towards separate releases and by extension towards the projects' split. Good point. There's nothing to stop us doing separate releases of sub-project components now. Doing so might help us find incompatibilities between the different components in a release line (2.x at the moment). Thanks, +Vinod On Tue, Jan 29, 2013 at 2:40 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Suresh. Adding back other *-dev lists. On Jan 29, 2013, at 1:58 PM, Suresh Srinivas wrote: +1 for a release with all the changes that are committed. That way it carries all the important bug fixes. So, rather than debate more, I had a brief chat with Suresh and Todd. Todd suggested calling the next release as hadoop-2.1.0-alpha to indicate the incompatibility a little better. This makes sense to me, as long as we are clear that we won't make any further *feature* releases in hadoop-2.0.x series (obviously we might be forced to do security/bug-fix release). We have been incorrectly using point releases to introduce features. Given there are many features in this release, calling it 2.1.0 instead of 2.0.3 makes sense. As you said, I am okay with the proposed plan as long as we do not lapse back to introducing new features in point releases meant for critical bugs. -- +Vinod Hortonworks Inc. http://hortonworks.com/
[jira] [Resolved] (MAPREDUCE-4850) Job recovery may fail if staging directory has been deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4850. -- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I ran test-patch and it came back clean. I just committed this. Job recovery may fail if staging directory has been deleted --- Key: MAPREDUCE-4850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4850 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Tom White Assignee: Tom White Fix For: 1.2.0 Attachments: MAPREDUCE-4850.patch, MAPREDUCE-4850.patch The job staging directory is deleted in the job cleanup task, which happens before the job-info file is deleted from the system directory (by the JobInProgress garbageCollect() method). If the JT shuts down between these two operations, then when the JT restarts and tries to recover the job, it fails since the job.xml and splits are no longer available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4850) Job recovery may fail if staging directory has been deleted
Tom White created MAPREDUCE-4850: Summary: Job recovery may fail if staging directory has been deleted Key: MAPREDUCE-4850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4850 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Tom White Assignee: Tom White The job staging directory is deleted in the job cleanup task, which happens before the job-info file is deleted from the system directory (by the JobInProgress garbageCollect() method). If the JT shuts down between these two operations, then when the JT restarts and tries to recover the job, it fails since the job.xml and splits are no longer available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
Tom White created MAPREDUCE-4824: Summary: Provide a mechanism for jobs to indicate they should not be recovered on restart Key: MAPREDUCE-4824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1 Affects Versions: 1.1.0 Reporter: Tom White Assignee: Tom White Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.) Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4632) Make sure MapReduce declares correct set of dependencies
Tom White created MAPREDUCE-4632: Summary: Make sure MapReduce declares correct set of dependencies Key: MAPREDUCE-4632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4632 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 2.0.0-alpha Reporter: Tom White This is the equivalent of HADOOP-8278 for MapReduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4567) Fix failing TestJobKillAndFail in branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White reopened MAPREDUCE-4567: -- Reverted and reopened along with MAPREDUCE-4488 - see that issue for details. Fix failing TestJobKillAndFail in branch-1 -- Key: MAPREDUCE-4567 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4567 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Fix For: 1.2.0 Attachments: MAPREDUCE-4567.patch This was introduced in MAPREDUCE-4488. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4610) Support deprecated mapreduce.job.counters.limit property in MR2
Tom White created MAPREDUCE-4610: Summary: Support deprecated mapreduce.job.counters.limit property in MR2 Key: MAPREDUCE-4610 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4610 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Tom White The property mapreduce.job.counters.limit was introduced in MAPREDUCE-1943, but the mechanism was changed in MAPREDUCE-901 where the property name was changed to mapreduce.job.counters.max without supporting the old name. We should deprecate but honour the old name to make it easier for folks to move from Hadoop 1 to Hadoop 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4567) Fix failing TestJobKillAndFail in branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4567. -- Resolution: Fixed Hadoop Flags: Reviewed I committed this to branch-1. Fix failing TestJobKillAndFail in branch-1 -- Key: MAPREDUCE-4567 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4567 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Tom White Assignee: Tom White Fix For: 1.2.0 Attachments: MAPREDUCE-4567.patch This was introduced in MAPREDUCE-4488. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4488) Port MAPREDUCE-463 (The job setup and cleanup tasks should be optional) to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4488. -- Resolution: Fixed Hadoop Flags: Reviewed I just committed this to branch-1. (Ran unit tests and test-patch successfully.) Port MAPREDUCE-463 (The job setup and cleanup tasks should be optional) to branch-1 --- Key: MAPREDUCE-4488 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4488 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, performance Affects Versions: 1.0.3 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-4488.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4257) Support fair-sharing option within a MR2 Capacity Scheduler queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4257. -- Resolution: Invalid Yes, I don't think we need this. Support fair-sharing option within a MR2 Capacity Scheduler queue - Key: MAPREDUCE-4257 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4257 Project: Hadoop Map/Reduce Issue Type: New Feature Components: capacity-sched, mrv2 Reporter: Tom White Assignee: Karthik Kambatla The fair scheduler can run jobs in a single pool (queue) in FIFO or fair share mode. In FIFO mode one job runs at a time, in priority order, while in fair share mode multiple jobs can run at the same time, and they share the capacity of the pool. This JIRA is to add the latter feature to Capacity Scheduler as an option - the default would remain FIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3837. -- Resolution: Fixed Fix Version/s: (was: 1.1.1) 1.2.0 I just committed this to branch-1. Thanks Mayank! Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job. Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0, 1.1.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 1.2.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837-3.patch, PATCH-HADOOP-1-MAPREDUCE-3837-4.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4337) Support multiple reducers in 'uber' jobs
Tom White created MAPREDUCE-4337: Summary: Support multiple reducers in 'uber' jobs Key: MAPREDUCE-4337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4337 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Tom White This is MAPREDUCE-434 for 'uber' jobs (jobs that run in the same container as the AM). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4299) Terasort hangs with MR2 FifoScheduler
Tom White created MAPREDUCE-4299: Summary: Terasort hangs with MR2 FifoScheduler Key: MAPREDUCE-4299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs. When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4252) MR2 job never completes with 1 pending task
Tom White created MAPREDUCE-4252: Summary: MR2 job never completes with 1 pending task Key: MAPREDUCE-4252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4252 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Tom White Assignee: Tom White This was found by ATM: bq. I ran a teragen with 1000 map tasks. Many task attempts failed, but after 999 of the tasks had completed, the job is now sitting forever with 1 task pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4230) Ensure framework counter group synchronization is correct for serialization/deserialization
Tom White created MAPREDUCE-4230: Summary: Ensure framework counter group synchronization is correct for serialization/deserialization Key: MAPREDUCE-4230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4230 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tom White Fix For: 2.0.0 FrameworkCounterGroup and FileSystemCounterGroup may be susceptible to a race outlined in https://issues.apache.org/jira/browse/MAPREDUCE-4226?focusedCommentId=13269657page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13269657 by Robert Joseph Evans. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4226) ConcurrentModificationException in FileSystemCounterGroup
Tom White created MAPREDUCE-4226: Summary: ConcurrentModificationException in FileSystemCounterGroup Key: MAPREDUCE-4226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4226 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Tom White Assignee: Tom White This was seen in a Hive job. I'll attach a failing test case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4217) Task commit waits for up to 3 seconds
Tom White created MAPREDUCE-4217: Summary: Task commit waits for up to 3 seconds Key: MAPREDUCE-4217 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4217 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.1 Reporter: Tom White Assignee: Tom White Following MAPREDUCE-2450, TaskReporter#stopCommunicationThread may wait up to 3 seconds (the value of PROGRESS_INTERVAL) before it returns. This can be noticeable for short running tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4217) Task commit waits for up to 3 seconds
[ https://issues.apache.org/jira/browse/MAPREDUCE-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4217. -- Resolution: Duplicate MAPREDUCE-3809 is the same. Thanks for pointing it out Siddharth. Task commit waits for up to 3 seconds - Key: MAPREDUCE-4217 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4217 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.1 Reporter: Tom White Assignee: Tom White Following MAPREDUCE-2450, TaskReporter#stopCommunicationThread may wait up to 3 seconds (the value of PROGRESS_INTERVAL) before it returns. This can be noticeable for short running tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4168) Support multiple network interfaces
Support multiple network interfaces --- Key: MAPREDUCE-4168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4168 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Tom White Umbrella jira to track the MapReduce side of HADOOP-8198. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4158) Port mapreduce.MapFileOutputFormat to branch-1
Port mapreduce.MapFileOutputFormat to branch-1 -- Key: MAPREDUCE-4158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4158 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 1.0.2 Reporter: Tom White Assignee: Tom White MapFileOutputFormat was missed in MAPREDUCE-3607. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3378) Create a single 'hadoop-mapreduce' Maven artifact
[ https://issues.apache.org/jira/browse/MAPREDUCE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3378. -- Resolution: Won't Fix I've opened HADOOP-8278 to track 1. HADOOP-8009 addressed 2. So I'm closing this JIRA now. Create a single 'hadoop-mapreduce' Maven artifact - Key: MAPREDUCE-3378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3378 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Tom White Attachments: MAPREDUCE-3378.patch In 0.23.0 there are multiple artifacts (hadoop-mapreduce-client-app, hadoop-mapreduce-client-common, hadoop-mapreduce-client-core, etc). It would be simpler for users to declare a dependency on hadoop-mapreduce (much like there's hadoop-common and hadoop-hdfs). (This would also be a step towards MAPREDUCE-2600.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4146) Support limits on task status string length and number of block locations in branch-2
Support limits on task status string length and number of block locations in branch-2 - Key: MAPREDUCE-4146 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4146 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White This brings MAPREDUCE-1943 to branch-2. Counter limits were introduced in MAPREDUCE-901. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4147) YARN should not have a compile-time dependency on HDFS
YARN should not have a compile-time dependency on HDFS -- Key: MAPREDUCE-4147 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4147 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-4147.patch YARN doesn't (and shouldn't) use any HDFS-specific APIs, so it should not declare HDFS as a compile-time dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4148) MapReduce should not have a compile-time dependency on HDFS
MapReduce should not have a compile-time dependency on HDFS --- Key: MAPREDUCE-4148 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tom White Assignee: Tom White MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
Reduce memory usage of counters due to non-static nested classes Key: MAPREDUCE-4138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Assignee: Tom White FrameworkCounter is a non-static nested class of FrameworkCounterGroup which means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2835) Make per-job counter limits configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-2835. -- Resolution: Fixed Fix Version/s: 1.1.0 Hadoop Flags: Reviewed I've just committed this. Make per-job counter limits configurable Key: MAPREDUCE-2835 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2835 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.204.0 Reporter: Tom White Assignee: Tom White Fix For: 1.1.0 Attachments: MAPREDUCE-2835.patch, MAPREDUCE-2835.patch The per-job counter limits introduced in MAPREDUCE-1943 are fixed, except for the total number allowed per job (mapreduce.job.counters.limit). It would be useful to make them all configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3935) Annotate Counters.Counter and Counters.Group as @Public
Annotate Counters.Counter and Counters.Group as @Public --- Key: MAPREDUCE-3935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3935 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White For clarity these inner classes should be marked as public stable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3936) Improve counter limits behaviour in 1.x
Improve counter limits behaviour in 1.x --- Key: MAPREDUCE-3936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Reporter: Tom White The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf instance to load the limits, which may throw an exception if the client limit is set to be lower than the limit on the cluster (perhaps because the cluster limit was raised from the default). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache
Allow symlinking in LocalJobRunner DistributedCache --- Key: MAPREDUCE-3871 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3871.patch Currently the LocalJobRunner doesn't create symlinks for files in the DistributedCache. It is safe to create symlinks if files of the same name don't exist. LocalJobRunner should also delete the symlinks when the job has completed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3858) Task attempt failure during commit results in task never completing
Task attempt failure during commit results in task never completing
---
Key: MAPREDUCE-3858
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3858
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Tom White
Assignee: Tom White
Priority: Critical
Attachments: MAPREDUCE-3858.patch
On a terasort job a task attempt failed during the commit phase. Another
attempt was rescheduled, but when it tried to commit it failed.
{noformat}
attempt_1329019187148_0083_r_000586_0 already given a go for committing the
task output, so killing attempt_1329019187148_0083_r_000586_1
{noformat}
The job hung as new attempts kept getting scheduled only to fail during commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3854) Reinstate environment variable tests in TestMiniMRChildTask
Reinstate environment variable tests in TestMiniMRChildTask --- Key: MAPREDUCE-3854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3854 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv2 Reporter: Tom White Fix For: 0.23.2 MAPREDUCE-3716 reinstated one of the tests in TestMiniMRChildTask, but there are two more which should be run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Priority: Blocker Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3684) LocalDistributedCacheManager does not shut down its thread pool
LocalDistributedCacheManager does not shut down its thread pool --- Key: MAPREDUCE-3684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3684 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White This was observed by running a Hive job in local mode. The job completed but the client process did not exit for 60 seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3636) Apply audience and stability annotations to classes in MapReduce for 1.x
Apply audience and stability annotations to classes in MapReduce for 1.x Key: MAPREDUCE-3636 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3636 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Port MAPREDUCE-1623 to branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3607) Port missing new API mapreduce lib classes to 1.x
Port missing new API mapreduce lib classes to 1.x - Key: MAPREDUCE-3607 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3607 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 1.0.0 Reporter: Tom White Assignee: Tom White There are a number of classes under mapreduce.lib that are not present in the 1.x series. Including these would help users and downstream projects using the new MapReduce API migrate to later versions of Hadoop in the future. A few examples of where this would help: * Sqoop uses mapreduce.lib.db.DBWritable and mapreduce.lib.input.CombineFileInputFormat (SQOOP-384). * Mahout uses mapreduce.lib.output.MultipleOutputs (MAHOUT-822). * HBase has a backport of mapreduce.lib.partition.InputSampler and TotalOrderPartitioner (in org.apache.hadoop.hbase.mapreduce.hadoopbackport) - it would be better if it used the ones in Hadoop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3467) Mavenizing har
[ https://issues.apache.org/jira/browse/MAPREDUCE-3467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3467. -- Resolution: Duplicate This was fixed in HADOOP-7810. Mavenizing har -- Key: MAPREDUCE-3467 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3467 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: John George Priority: Critical As part of mapreduce mavenization, har should also be mavenized and added to maven repo -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3595) Add missing TestCounters#testCounterValue test from branch 1 to 0.23
Add missing TestCounters#testCounterValue test from branch 1 to 0.23 Key: MAPREDUCE-3595 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3595 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3595.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3542) Support FileSystemCounter legacy counter group name for compatibility
Support FileSystemCounter legacy counter group name for compatibility --- Key: MAPREDUCE-3542 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3542 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White The group name changed from FileSystemCounter to org.apache.hadoop.mapreduce.FileSystemCounter, but we should support the old one for compatibility's sake. This came up in PIG-2347. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3527) Fix minor API incompatibilities between 1.0 and 0.23
Fix minor API incompatibilities between 1.0 and 0.23 Key: MAPREDUCE-3527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3527 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3428) MR AppMaster CLASSPATH is dependent on the compile-time environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3428. -- Resolution: Duplicate MR AppMaster CLASSPATH is dependent on the compile-time environment Key: MAPREDUCE-3428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3428 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ahmed Radwan Labels: bigtop The CLASSPATH for the MapReduce Application master is set using compile time path information, which is typically different from run-time. This will cause failure when running on different environments. Specifically, the YarnRunner, and as part ApplicationSubmissionContext creation, sets the classpath for the application master using MRApps.setClasspath(environment), and then the setMRFrameworkClasspath(..) method uses compile time path information present in the mrapp-generated-classpath file (created at compile-time). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3479) JobClient#getJob cannot find local jobs
JobClient#getJob cannot find local jobs --- Key: MAPREDUCE-3479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3479 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White The problem is that JobClient#submitJob doesn't pass the Cluster object to Job for the submission process, which means that two Cluster objects and two LocalJobRunner objects are created. LocalJobRunner keeps an instance map of job IDs to Jobs, and when JobClient#getJob is called the LocalJobRunner with the unpopulated map is used which results in the job not being found. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3416) Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
Rename {start,stop}-all.sh to {start,stop}-yarn.sh for consistency with HDFS
Key: MAPREDUCE-3416
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3416
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Tom White
There are already {start,stop}-all.sh scripts for starting and stopping all
Hadoop daemons which conflict with the YARN {start,stop}-all.sh scripts. The
latter should be renamed {start,stop}-yarn.sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3422) Counter display names are not being picked up
Counter display names are not being picked up - Key: MAPREDUCE-3422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3422 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White When running a job I see MAP_INPUT_RECORDS rather than Map input records for the counter name. To fix this the resource bundle properties files need to be moved to the src/main/resources tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3378) Create a single 'hadoop-mapreduce' Maven artifact
Create a single 'hadoop-mapreduce' Maven artifact - Key: MAPREDUCE-3378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3378 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.23.0 Reporter: Tom White In 0.23.0 there are multiple artifacts (hadoop-mapreduce-client-app, hadoop-mapreduce-client-common, hadoop-mapreduce-client-core, etc). It would be simpler for users to declare a dependency on hadoop-mapreduce (much like there's hadoop-common and hadoop-hdfs). (This would also be a step towards MAPREDUCE-2600.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: hadoop-0.23.0 jiras
Hi Arun, A couple more issues that I think are blockers: HADOOP-7782 (Aggregate project javadocs) should be a blocker since otherwise private APIs are published as a part of the public API. MAPREDUCE-3237 (Move LocalJobRunner to hadoop-mapreduce-client-core module) is also a blocker since without it I don't think it's possible to run local jobs. Also, what do you want to do about the documentation? Some is in Forrest and some is in APT. Here are the current 0.23 blockers: https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hiderequestId=12317667 Cheers, Tom On Mon, Oct 31, 2011 at 12:11 AM, Arun C Murthy a...@hortonworks.com wrote: Done. Ping me if you find anything astray. thanks, Arun On Oct 30, 2011, at 9:18 PM, Arun C Murthy wrote: FYI - I'm starting to change 'fix version' for all non-blockers and do a final pass on open blockers. I'll send a note out when done. thanks, Arun
[jira] [Created] (MAPREDUCE-3237) Move LocalJobRunner to hadoop-mapreduce-client-core module
Move LocalJobRunner to hadoop-mapreduce-client-core module -- Key: MAPREDUCE-3237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3237 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Fix For: 0.23.0 LocalJobRunner works independently of MR1 (jobtracker and tasktrackers) and MR2 (YARN). The MR1 directory is being kept around only to support unit tests, so LocalJobRunner should be moved out to somewhere more permanent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
Publish Yarn and MapReduce artifacts to Maven snapshot repository - Key: MAPREDUCE-3003 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Tom White Assignee: Tom White Currently this is failing since no distribution management section is defined in the POM. https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problem getting the code from SVN
On Sun, Aug 21, 2011 at 11:13 AM, Konstantin Boudnik c...@apache.org wrote: Seems like it has been broken in the recent re-shuffle of the workspace due to the maven changes. This external now lives under hadoop-common, apparently. It's actually still under common, however it's no longer needed for HDFS since it has been replaced by test-patch.sh in dev-support. I'll file a ticket to remove the externals. On Sun, Aug 21, 2011 at 07:20PM, Praveen Sripati wrote: Hi, When I try to get the code from svn, I get the below error. svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/ A trunk/hadoop-mapreduce/bin/mapred-config.sh A trunk/hadoop-mapreduce/bin/stop-mapred.sh A trunk/hadoop-mapreduce/bin/mapred A trunk/hadoop-mapreduce/bin/start-mapred.sh A trunk/hadoop-project A trunk/hadoop-project/pom.xml U trunk Fetching external item into 'trunk/hadoop-hdfs/src/test/bin' svn: warning: OPTIONS of ' https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin': Could not resolve hostname `svn.apache.org': No address associated with hostname (https://svn.apache.org) Is this a DNS issue? Cheers, Tom svn: warning: Error handling externals definition for 'trunk/hadoop-mapreduce/src/test/bin': svn: warning: OPTIONS of ' https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin': Could not resolve hostname `svn.apache.org': No address associated with hostname (https://svn.apache.org) Checked out revision 1159979. Thanks, Praveen
Re: Mavenizing the HDFS build
HDFS-2096 is now committed to trunk. The instructions for building HDFS can be found in the top-level BUILDING.txt file. I added a script to https://issues.apache.org/jira/browse/HADOOP-7500 to help with migrating HDFS patches to the new layout. There are a few follow-up patches that need doing soon (e.g. HADOOP-7498, HADOOP-7496, MAPREDUCE-2856), but these shouldn't stop folks from doing development as usual. Thanks to everyone who helped with this! Cheers, Tom On Thu, Aug 18, 2011 at 11:30 AM, Tom White t...@cloudera.com wrote: Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Re: Mavenizing the HDFS build
Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Re: [VOTE] Merge MR-279 to trunk.
Excellent work. +1 Tom On Tue, Aug 16, 2011 at 2:14 PM, Mahadev Konar maha...@hortonworks.com wrote: Hi all, We are excited to let you know that we have MR-279 ready to be merged to trunk. I have uploaded necessary details on https://issues.apache.org/jira/browse/MAPREDUCE-279. Please take a look and vote. Clearly I am +1 on it. thanks mahadev
Re: Mavenizing the HDFS build
Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
[jira] [Created] (MAPREDUCE-2835) Make per-job counter limits configurable
Make per-job counter limits configurable Key: MAPREDUCE-2835 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2835 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.204.0 Reporter: Tom White Assignee: Tom White Fix For: 0.20.205.0 The per-job counter limits introduced in MAPREDUCE-1943 are fixed, except for the total number allowed per job (mapreduce.job.counters.limit). It would be useful to make them all configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2741) Make ant build system work with hadoop-common JAR generated by Maven
[ https://issues.apache.org/jira/browse/MAPREDUCE-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-2741. -- Resolution: Fixed Hadoop Flags: [Reviewed] I've committed this. Thanks, Alejandro. Make ant build system work with hadoop-common JAR generated by Maven Key: MAPREDUCE-2741 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2741 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0 Attachments: MAPRED-2741.patch, MAPREDUCE-2741.patch Some tweaks must be done in MAPRED its contribs ivy configuration to work with HADOOP-6671. This wil be a temporary fix until MAPRED is mavenized. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: MR1 next steps
+1 for #2 as long as the user-level MR API remains compatible. Cheers, Tom On Thu, Jul 7, 2011 at 9:58 AM, Eli Collins e...@cloudera.com wrote: Hey gang, Had some discussion about what to do with MR1 with Arun at the summit, wanted to move it on-list.. Was thinking we should sort these out some on mr-dev before discussing/announcing a decision on general. The question is, now that we'll soon have MR2 merged (hurray!), to what extent do we ant to support MR1? By MR1 I mean the JT and TT, not the old MR API, which MR2 supports. Ie this isn't about job API compatibility it's about implementation compatibility (eg existing systems which may depend on JT/TT interfaces like metrics). Here are the options as I see them: 1. Do nothing. MR1 will continue to be a regression, both in terms of features and stability, against the MR in 203. Eg, MR1 in trunk still doesn't support security. We would continue to recommend people use MR1 from 20 (and MR2 from 23). Unclear what the value of having MR1 in trunk in this shape is. 2. Remove the MR1 code from trunk/23, and just support MR2 in 23. People who want MR1 can use the current stable release (which, per option 1, we would recommend even if we left the code in as is). 3. Get MR1 in trunk in shape comparable to MR in 203. This preserves the additional changes (to JT/TT at least) that have been added in trunk since 0.20. Not clear if anyone would want to invest the considerable effort this would take given that we have MR2 now (and existing releases). 4. Put the MR1 code from 203 into trunk. This overwrites the changes added to trunk not in 203, and would require some integration, however it would give us a solid MR1 implementation that could be used in the same release as MR2. It would be an incompatible change wrt 21/22, however would be compatible in the sense that there are now both valid MR1 and MR2 options in a single release. I think #2 makes the most sense. From a developer perspective, MR2 is good stuff, there's no need for us to maintain two implementations in trunk/23 since we're already maintaining MR1 in the current releases. I'm skeptical that anyone would volunteer to do #3 (lot of work, unclear gain) or #4 (we already maintain MR1 elsewhere). This allows us to focus energy on MR2 instead of investing in MR1 (eg MR-2178, which hasn't made much progress for ages). From a user perspective, MR2 preserves Job compatibility, so it should just programs that talk to the JT/TT that are affected. MR2 is a little harder to run out-of-the-box, however we can fix that and we don't recommend people use MR1 from 21/22/trunk anyway. Thoughts? Thanks, Eli
[jira] [Created] (MAPREDUCE-2638) Create a simple stress test for the fair scheduler
Create a simple stress test for the fair scheduler -- Key: MAPREDUCE-2638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2638 Project: Hadoop Map/Reduce Issue Type: Test Components: contrib/fair-share Reporter: Tom White Assignee: Tom White This would be a test that runs against a cluster, typically with settings that allow preemption to be exercised. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2455) Remove deprecated JobTracker.State in favour of JobTrackerStatus
Remove deprecated JobTracker.State in favour of JobTrackerStatus Key: MAPREDUCE-2455 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2455 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Tom White Assignee: Tom White Fix For: 0.23.0 MAPREDUCE-2337 deprecated getJobTrackerState() on ClusterStatus, this issue is to remove the getter (in favour of getJobTrackerStatus(), which will remain) so there is no longer a direct dependency of the public API on JobTracker. This is for MAPREDUCE-1638. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2421) Remove JobHistory's dependency on JobTracker
Remove JobHistory's dependency on JobTracker Key: MAPREDUCE-2421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2421 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Reporter: Tom White Assignee: Tom White The direct dependency can be removed by introducing a callback inteface. This is for MAPREDUCE-1638. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2400) Remove Cluster's dependency on JobTracker
Remove Cluster's dependency on JobTracker - Key: MAPREDUCE-2400 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2400 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Reporter: Tom White Assignee: Tom White Introduce a factory using ServiceLoader to remove the direct dependency. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAPREDUCE-2392) TaskTracker shutdown in the tests sometimes take 60s
[
https://issues.apache.org/jira/browse/MAPREDUCE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White resolved MAPREDUCE-2392.
--
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
I've just committed this.
TaskTracker shutdown in the tests sometimes take 60s
Key: MAPREDUCE-2392
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2392
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
Fix For: 0.22.0
Attachments: MAPREDUCE-2392.patch
There are a lot of the following in the test logs:
{noformat}
2011-03-16 13:47:02,267 INFO mapred.TaskTracker
(TaskTracker.java:shutdown(1275)) - Shutting down StatusHttpServer
2011-03-16 13:48:02,349 ERROR mapred.TaskTracker
(TaskTracker.java:offerService(1609)) - Caught exception:
java.io.IOException: Call to localhost/127.0.0.1:57512 failed on local
exception: java.nio.channels.ClosedByInterruptException
{noformat}
Note there is over one minute between the first line and the second.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2351) mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI
mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI Key: MAPREDUCE-2351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2351 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Assignee: Tom White Currently, mapred.job.tracker.history.completed.location is resolved relative to the default filesystem. If not set it defaults to history/done in the local log directory. There is no way to set it to another local filesystem location (with a file:// URI) or an arbitrary Hadoop filesystem. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2338) Deprecate field selection methods on JobConf
Deprecate field selection methods on JobConf Key: MAPREDUCE-2338 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2338 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White KeyFieldBasedComparator and KeyFieldBasedPartitioner are libraries so should not appear as getters and setters in JobConf. For the new API there are getters and setters on KeyFieldBasedComparator and KeyFieldBasedPartitioner themselves - we should add the equivalent to the old API and deprecate the references in JobConf. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2337) Remove dependence of public MapReduce API on classes in server package
Remove dependence of public MapReduce API on classes in server package -- Key: MAPREDUCE-2337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2337 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Assignee: Tom White Cluster#getJobTrackerState() returns a org.apache.hadoop.mapreduce.server.jobtracker.State enum, which makes the API in o.a.h.mapreduce have a dependency on the server package. It would be better to make the public API self-contained by using an equivalent enum in the Cluster class. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2282) MapReduce tests don't compile following HDFS-1561
MapReduce tests don't compile following HDFS-1561 - Key: MAPREDUCE-2282 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2282 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tom White Priority: Blocker TestMRServerPorts depends on TestHDFSServerPorts which was changed by HDFS-1561, resulting in a compilation failure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2273) TaskLogServlet does not set content type
[ https://issues.apache.org/jira/browse/MAPREDUCE-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-2273. -- Resolution: Duplicate TaskLogServlet does not set content type Key: MAPREDUCE-2273 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2273 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Tom White Priority: Blocker TaskLogServlet has never set the content type, even though it can serve plain text or HTML, but since HADOOP-7093 the problem has been highlighted since it serves HTML as text/plain (before it was serving plain text as text/html). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2086) CHANGES.txt does not reflect the release of version 0.21.0.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-2086. -- Resolution: Fixed Assignee: Tom White I've fixed this. CHANGES.txt does not reflect the release of version 0.21.0. --- Key: MAPREDUCE-2086 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2086 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Assignee: Tom White Fix For: 0.21.1 CHANGES.txt should show the release date for 0.21.0 and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2092) Trunk can't be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White resolved MAPREDUCE-2092.
--
Resolution: Fixed
HADOOP-6951 has now been reverted and I confirmed MapReduce trunk now compiles.
Sorry for the noise.
Trunk can't be compiled
---
Key: MAPREDUCE-2092
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2092
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Reporter: Konstantin Boudnik
Compilation of the trunk is broken because of an attempt to call
{{ServiceAuthorizationManager.refresh}} from a static content.
0.21 branch seems to be Ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2014) Remove task-controller from 0.21 branch
Remove task-controller from 0.21 branch --- Key: MAPREDUCE-2014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2014 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Tom White Assignee: Tom White Fix For: 0.21.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1870) Harmonize MapReduce JAR library versions with Common and HDFS
Harmonize MapReduce JAR library versions with Common and HDFS - Key: MAPREDUCE-1870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1870 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 MapReduce part of HADOOP-6800. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: building 0.21
Perhaps you're not getting up-to-date libraries? Try common ant clean jar mvn-install hdfs ant veryclean jar mvn-install -Dresolvers=internal mapreduce ant veryclean jar -Dresolvers=internal This works for me with the 0.21 branches. Cheers, Tom On Tue, Jun 8, 2010 at 9:48 AM, Torsten Curdt tcu...@apache.org wrote: I've build all the various dependencies hadoop-commons (branch 0.21) hadoop-hdfs (branch 0.21) hadoop (trunk == 0.21?? there is no 0.21 branch) and then hadoop-mapreduce (branch 0.21) Using it I get the following exeception: Exception in thread main java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecation(Ljava/lang/String;[Ljava/lang/String;)V at org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:49) at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:40) at org.apache.hadoop.mapreduce.Cluster.clinit(Cluster.java:64) What's going on? cheers -- Torsten
[jira] Resolved: (MAPREDUCE-1810) 0.21 build is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-1810. -- Fix Version/s: 0.21.0 Resolution: Fixed I've just committed this (to the 0.21 branch). 0.21 build is broken Key: MAPREDUCE-1810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1810 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0 Reporter: Sharad Agarwal Assignee: Tom White Fix For: 0.21.0 Attachments: MAPREDUCE-1810.patch, MAPREDUCE-1810.patch /src/java/org/apache/hadoop/mapred/AdminOperationsProtocol.java:31: cannot find symbol [javac] symbol : method value() [javac] location: @interface org.apache.hadoop.security.KerberosInfo [javac] @KerberosInfo(MRJobConfig.JOB_JOBTRACKER_ID) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1751) Change MapReduce to depend on Hadoop 'common' artifacts instead of 'core'
[ https://issues.apache.org/jira/browse/MAPREDUCE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-1751. -- Hadoop Flags: [Reviewed] Assignee: Tom White Resolution: Fixed I've just committed this. Change MapReduce to depend on Hadoop 'common' artifacts instead of 'core' - Key: MAPREDUCE-1751 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1751 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-1751.patch This is the MapReduce part of HADOOP-6404. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1789) MapReduce trunk fails to compile following HADOOP-6600
MapReduce trunk fails to compile following HADOOP-6600 -- Key: MAPREDUCE-1789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1789 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 A few classes need updating following the change to KerberosInfo introduced in HADOOP-6600 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Minutes: Hadoop Contributor Meeting 05/06/2010
Not sure why my attachment didn't make it to the list. Anyway, I've posted Arun's notes on the wiki at http://wiki.apache.org/hadoop/HadoopContributorsMeeting20100506, and included the content of my slide there. (Attachments on the wiki have been disabled - as of today apparently, see SVN commit r775220 - so I wasn't able to post the slide there either.) Tom On Fri, May 7, 2010 at 9:36 AM, Tom White t...@cloudera.com wrote: Here's my (single) slide about the 0.21 release. Tom On Thu, May 6, 2010 at 5:38 PM, Arun C Murthy acmur...@gmail.com wrote: # Shared goals - Hadoop is HDFS Map-Reduce in this context of this set of slides # Priorities * Yahoo - Correctness - Availability: Not the same as high-availability (6 9s. etc.) i.e. SPOFs - API Compatibility - Scalability - Operability - Performance - Innovation * Cloudera - Test coverage, api coverage - APL Licensed codec (lzo replacement) - Security - Wire compatibility - Cluster-wide resource availability - New apis (FileContext, MR Context Objs.), documentation of their advantages - HDFS to better support non-MR use-cases - Cluster metrics hooks - MR modularity (package) * Facebook - Correctness - Availability, High Availability, Failover, Continuous Availability - Scalability # Bar for patches/features keeps going higher as the project matures - Build consensus (e.g. Python Enhancement Process, JSR etc.) - Run/test on your own to prove the concept/feature or branch and finish - Early versions of libraries should be started outside of the project (github etc.) e.g. input-formats, compression-codecs etc. - github for all the above - Prune contrib # Maven for packaging # Tom: hadoop-0.21 (Tom - can you please post your slides? Thanks!) # Owen: Release Manager (see slides) # Agenda for next meeting - Eli: Hadoop Enhancement Process (modelled on PEP?) - Branching strategies: Development Models Arun
Re: Minutes: Hadoop Contributor Meeting 05/06/2010
Here's my (single) slide about the 0.21 release. Tom On Thu, May 6, 2010 at 5:38 PM, Arun C Murthy acmur...@gmail.com wrote: # Shared goals - Hadoop is HDFS Map-Reduce in this context of this set of slides # Priorities * Yahoo - Correctness - Availability: Not the same as high-availability (6 9s. etc.) i.e. SPOFs - API Compatibility - Scalability - Operability - Performance - Innovation * Cloudera - Test coverage, api coverage - APL Licensed codec (lzo replacement) - Security - Wire compatibility - Cluster-wide resource availability - New apis (FileContext, MR Context Objs.), documentation of their advantages - HDFS to better support non-MR use-cases - Cluster metrics hooks - MR modularity (package) * Facebook - Correctness - Availability, High Availability, Failover, Continuous Availability - Scalability # Bar for patches/features keeps going higher as the project matures - Build consensus (e.g. Python Enhancement Process, JSR etc.) - Run/test on your own to prove the concept/feature or branch and finish - Early versions of libraries should be started outside of the project (github etc.) e.g. input-formats, compression-codecs etc. - github for all the above - Prune contrib # Maven for packaging # Tom: hadoop-0.21 (Tom - can you please post your slides? Thanks!) # Owen: Release Manager (see slides) # Agenda for next meeting - Eli: Hadoop Enhancement Process (modelled on PEP?) - Branching strategies: Development Models Arun
[jira] Created: (MAPREDUCE-1751) Change MapReduce to depend on Hadoop 'common' artifacts instead of 'core'
Change MapReduce to depend on Hadoop 'common' artifacts instead of 'core' - Key: MAPREDUCE-1751 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1751 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Tom White Priority: Blocker Fix For: 0.21.0 This is the MapReduce part of HADOOP-6404. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1735) Un-deprecate the old MapReduce API in the 0.21 branch
Un-deprecate the old MapReduce API in the 0.21 branch - Key: MAPREDUCE-1735 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1735 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Priority: Blocker Fix For: 0.21.0 This issue is to un-deprecate the old MapReduce API (in o.a.h.mapred) in the next 0.21 release, as discussed at http://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg01833.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Un-deprecate the old MapReduce API?
It sounds like there's no strong objection to un-deprecating the old
API in 0.20 - I'll create a patch for this (see
https://issues.apache.org/jira/browse/MAPREDUCE-1734).
0.21 is less clear cut. However, if the new API were marked as
Evolving, then it's odd, to say the least, if the old API were
deprecated since it would send a confusing message to users. There
seems to be consensus that the new API is Evolving (please comment on
https://issues.apache.org/jira/browse/MAPREDUCE-1623 to discuss
whether all of the new API should be marked Evolving, which the latest
patch does). Indeed, the new API hasn't seen widespread use yet, so it
still seems premature to deprecate the old API in 0.21. I've opened
https://issues.apache.org/jira/browse/MAPREDUCE-1735 where we can
discuss this particular case further.
Cheers,
Tom
On Fri, Apr 23, 2010 at 9:21 AM, Alan Gates ga...@yahoo-inc.com wrote:
I don't have any issue with un-deprecating the old APIs. I agree if changes
are needed it's better to mark the new APIs to reflect it. I just hope
those changes can be kept as backward compatible as possible. In particular
with Job, Pig uses that in some of it's APIs that it has declared stable
(LoadFunc, StoreFunc).
Alan.
On Apr 22, 2010, at 11:30 PM, Arun C Murthy wrote:
Alan,
On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:
Speaking for one power user (Pig) that did move to the new APIs, moving
that interface to evolving is a little unsettling. Is there a feel for how
much the new API is going to change?
The intent isn't to mark the 'new' apis as 'Evolving' to change them
willy-nilly... please don't read it so!
This is just a pragmatic proposal to reflect that the 'old' apis will, for
lack of stabilization of new apis, be supported.
Given that, the new apis could mostly be stable, but for Job and Cluster -
is that reasonable? This will ensure we send the right message all concerned
regarding stability of o.a.h.mapreduce.{Mapper|Reducer|...}. Thoughts?
Arun
Alan.
[jira] Created: (MAPREDUCE-1681) MapReduce API compatibility
MapReduce API compatibility --- Key: MAPREDUCE-1681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1681 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, documentation Reporter: Tom White Priority: Blocker Fix For: 0.21.0 This is an umbrella issue to document and test MapReduce API compatibility across releases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1650) Exclude Private elements from generated MapReduce Javadoc
Exclude Private elements from generated MapReduce Javadoc - Key: MAPREDUCE-1650 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1650 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Tom White Exclude elements annotated with InterfaceAudience.Private or InterfaceAudience.LimitedPrivate from Javadoc and JDiff. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1637) Create a test for API compatibility between releases
Create a test for API compatibility between releases Key: MAPREDUCE-1637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1637 Project: Hadoop Map/Reduce Issue Type: Test Components: build, test Reporter: Tom White Fix For: 0.21.0 We should have an automated test (or a set of tests) for checking that programs written against an old version of the API still run with a newer version. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1638) Divide MapReduce into API and implementation source trees
Divide MapReduce into API and implementation source trees - Key: MAPREDUCE-1638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Reporter: Tom White Assignee: Tom White I think it makes sense to separate the MapReduce source into public API and implementation trees. The public API could be broken further into kernel and library trees. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1625) Improve grouping of packages in Javadoc
Improve grouping of packages in Javadoc --- Key: MAPREDUCE-1625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1625 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Tom White There are a couple of problems with the current Javadoc: * The main MapReduce package documentation on the index page appears under Other Packages below the fold. * Some contrib classes and packages are interspersed in the main MapReduce documentation, which is very confusing for users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1626) Publish Javadoc for all contrib packages with user-facing APIs
Publish Javadoc for all contrib packages with user-facing APIs -- Key: MAPREDUCE-1626 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1626 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Some packages don't appear in the Javadoc. E.g. MRUnit, Vertica. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1592) Generate Eclipse's .classpath file from Ivy config
Generate Eclipse's .classpath file from Ivy config -- Key: MAPREDUCE-1592 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1592 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Tom White Assignee: Tom White MapReduce companion issue for HADOOP-6407. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1507) The old MapReduce API is only partially deprecated
The old MapReduce API is only partially deprecated -- Key: MAPREDUCE-1507 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1507 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tom White Assignee: Tom White Not all of the old API is currently marked as deprecated. E.g. org.apache.hadoop.mapred.OutputFormat is deprecated, but org.apache.hadoop.mapred.FileOutputFormat isn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1511) Examples should not use deprecated APIs
Examples should not use deprecated APIs --- Key: MAPREDUCE-1511 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1511 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission Reporter: Tom White Assignee: Tom White MAPREDUCE-777 deprecated some APIs which are still being used by the examples. This issue is to fix the examples so they use the replacements. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1452) Add a low-level MapReduce API
Add a low-level MapReduce API - Key: MAPREDUCE-1452 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1452 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Tom White Fix For: 0.22.0 Add an API to MapReduce that operates at the raw bytes level. The existing (object-based) MapReduce APIs would be implemented on top of the raw API, and in future it will be easier to add new APIs (like MAPREDUCE-1183) and higher-level abstractions on MapReduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1453) Enforce the distinction between MapReduce kernel and library code
Enforce the distinction between MapReduce kernel and library code - Key: MAPREDUCE-1453 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1453 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Fix For: 0.22.0 Currently MapReduce kernel and library code are in the same source tree which makes it possible to inadvertently introduce dependencies on the library by the kernel. With MAPREDUCE-1452 it makes sense to enforce the compile-time dependencies (library depends on kernel, but not vice versa), by having two source trees. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1411) Infer MapReduce types where possible
Infer MapReduce types where possible Key: MAPREDUCE-1411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1411 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Tom White Assignee: Tom White Currently users must specify the output types of the map and reduce if they are different to the default (LongWritable keys, Text values). In many cases this information is available to the system from the user-supplied mapper and reducer, so it would be nice if the user didn't have to specify them in two places. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1349) Create jira component - contrib/index
[ https://issues.apache.org/jira/browse/MAPREDUCE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-1349. -- Resolution: Fixed Assignee: Tom White Fixed. Create jira component - contrib/index -- Key: MAPREDUCE-1349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1349 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Kay Kay Assignee: Tom White Can we have a component - contrib/index for issues related to contrib/index (in jira ) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1348) Package org.apache.hadoop.blockforensics does not match directory name
Package org.apache.hadoop.blockforensics does not match directory name -- Key: MAPREDUCE-1348 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1348 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Tom White BlockSearch is in the package org.apache.hadoop.blockforensics, but in the source directory org/apache/hadoop/block_forensics. While javac doesn't seem to mind about this mismatch, Eclipse treats it as an error. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: mapreduce trunk is broken
This is due to https://issues.apache.org/jira/browse/HADOOP-5901, which I've now reverted. I just did svn up; ant veryclean compile and managed to build successfully. Cheers, Tom On Tue, Dec 15, 2009 at 11:24 AM, Zheng Shao zs...@facebook.com wrote: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/194/artifact/trunk/patchprocess/trunkReleaseAuditWarnings.txt compile-mapred-test: [mkdir] Created dir: /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/test/mapred/classes [mkdir] Created dir: /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/test/mapred/testjar [mkdir] Created dir: /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/test/mapred/testshell [javac] Compiling 237 source files to /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/test/mapred/classes [javac] /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/src/test/mapred/org/apache/hadoop/mapred/MiniMRCluster.java:342: unreported exception java.io.IOException; must be caught or declared to be thrown [javac] FileSystem.setDefaultUri(result, namenode); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 1 error BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build.xml:487: Compile failed; see the compiler error output for details. Total time: 1 minute 16 seconds Zheng
[jira] Resolved: (MAPREDUCE-690) Sqoop's test hive script needs to be executable
[ https://issues.apache.org/jira/browse/MAPREDUCE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-690. - Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] +1 I've just committed this. Thanks Aaron! Sqoop's test hive script needs to be executable - Key: MAPREDUCE-690 URL: https://issues.apache.org/jira/browse/MAPREDUCE-690 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Fix For: 0.21.0 The testdata/hive/bin/hive script needs to be chmod +x so that unit tests can run it. This needs to be set with an svn property. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
