Re: [VOTE] Release Apache Hadoop 2.6.0
Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary condition (and should not cause issues when for files with replication factor 3), we should perhaps target this into 2.6.1 and not block this release. Thoughts? On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: +0(non-binding) HDFS-6833 is critical issue for us - could you help us to merge it into 2.6? Thanks, - Tsuyoshi On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah hit...@apache.org wrote: +1 (binding) Built Hadoop from source, compiled Tez against the hadoop jars pushed to staging repo and ran a few example Tez jobs on a single node cluster. — HItesh On Nov 13, 2014, at 3:08 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- - Tsuyoshi -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
+1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: - Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. - Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Thu, Nov 13, 2014 at 3:08 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to turn off the feature does not change the compatibility story. I'm willing to devote time to help review these JIRAs and kick the tires on the APIs, but my point above was that I'm not sure it'd all be done by the end of the week. Testing might also reveal additional changes that need to be made, which also might not happen by end-of-week. I guess the question before us is if we're comfortable putting something in branch-2.6 and then potentially adding API changes after. I'm okay with that as long as we're all aware that this might happen. Arun, as RM is this cool with you? Again, I like this feature and I'm fine with it's inclusion, just a heads up that we might need some extra time to finalize things before an RC can be cut. Thanks, Andrew On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze s29752-hadoop...@yahoo.com.invalid wrote: Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It is not easy to get everything right. For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature. It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS. When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories. Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before. As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2. Regards, Tsz-Wo On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic
Re: Thinking ahead to hadoop-2.6
I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Updates on migration to git
Karthik, I would like to see detailed information on how this migration will be done, how it will affect the existing project and commit process. This should be done in a document that can be reviewed instead of in an email thread on an ad-hoc basis. Was there any voting on this in PMC and should we have a vote to ensure everyone is one the same page on doing this and how to go about it? Regards, Suresh On Tue, Aug 26, 2014 at 9:17 AM, Karthik Kambatla ka...@cloudera.com wrote: Last I heard, the import is still going on and appears closer to getting done. Thanks for your patience with the migration. I ll update you as and when there is something. Eventually, the git repo should be at the location in the wiki. On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for bringing these points up, Zhijie. By the way, a revised How-to-commit wiki is at: https://wiki.apache.org/hadoop/HowToCommitWithGit . Please feel free to make changes and improve it. On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen zs...@hortonworks.com wrote: Do we have any convention about user.name and user.email? For example, we'd like to use @apache.org for the email. May be, we can ask people to use project-specific configs here and use their real name and @apache.org address. Is there any downside to letting people use their global values for these configs? Moreover, do we want to use --author=Author Name em...@address.com when committing on behalf of a particular contributor? Fetching the email-address is complicated here. Should we use the contributor's email from JIRA? What if that is not their @apache address? On Mon, Aug 25, 2014 at 9:56 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for your input, Steve. Sorry for sending the email out that late, I sent it as soon as I could. On Mon, Aug 25, 2014 at 2:20 AM, Steve Loughran ste...@hortonworks.com wrote: just caught up with this after some offlininess...15:48 PST is too late for me. I'd be -1 to a change to master because of that risk that it does break existing code -especially people that have trunk off the git mirrors and automated builds/merges to go with it. Fair enough. It makes sense to leave it as trunk, unless someone is against it being trunk. master may be viewed as the official git way, but it doesn't have to be. For git-flow workflows (which we use in slider) master/ is for releases, develop/ for dev. On 24 August 2014 02:31, Karthik Kambatla ka...@cloudera.com wrote: Couple of things: 1. Since no one expressed any reservations against doing this on Sunday or renaming trunk to master, I ll go ahead and confirm that. I think that serves us better in the long run. 2. Arpit brought up the precommit builds - we should definitely fix them as soon as we can. I understand Giri maintains those builds, do we have anyone else who has access in case Giri is not reachable? Giri - please shout out if you can help us with this either on Sunday or Monday. Thanks Karthik On Fri, Aug 22, 2014 at 3:50 PM, Karthik Kambatla ka...@cloudera.com wrote: Also, does anyone know what we use for integration between JIRA and svn? I am assuming svn2jira. On Fri, Aug 22, 2014 at 3:48 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, For the SCM migration, feel free to follow https://issues.apache.org/jira/browse/INFRA-8195 Most of this is planned to be handled this Sunday. As a result, the subversion repository would be read-only. If this is a major issue for you, please shout out. Daniel Gruno, the one helping us with the migration, was asking if we are open to renaming trunk to master to better conform to git lingo. I am tempted to say yes, but wanted to check. Would greatly appreciate any help with checking the git repo has everything. Thanks Karthik -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie
Re: Updates on migration to git
:) I missed that voting thread. Thanks Karthik! Arpit Agarwal also told me offline that commit process has been updated - https://wiki.apache.org/hadoop/HowToCommit and git setup also has also been documented - https://www.apache.org/dev/git.html. Thanks Arpit! On Tue, Aug 26, 2014 at 11:41 AM, Suresh Srinivas sur...@hortonworks.com wrote: Karthik, I would like to see detailed information on how this migration will be done, how it will affect the existing project and commit process. This should be done in a document that can be reviewed instead of in an email thread on an ad-hoc basis. Was there any voting on this in PMC and should we have a vote to ensure everyone is one the same page on doing this and how to go about it? Regards, Suresh On Tue, Aug 26, 2014 at 9:17 AM, Karthik Kambatla ka...@cloudera.com wrote: Last I heard, the import is still going on and appears closer to getting done. Thanks for your patience with the migration. I ll update you as and when there is something. Eventually, the git repo should be at the location in the wiki. On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for bringing these points up, Zhijie. By the way, a revised How-to-commit wiki is at: https://wiki.apache.org/hadoop/HowToCommitWithGit . Please feel free to make changes and improve it. On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen zs...@hortonworks.com wrote: Do we have any convention about user.name and user.email? For example, we'd like to use @apache.org for the email. May be, we can ask people to use project-specific configs here and use their real name and @apache.org address. Is there any downside to letting people use their global values for these configs? Moreover, do we want to use --author=Author Name em...@address.com when committing on behalf of a particular contributor? Fetching the email-address is complicated here. Should we use the contributor's email from JIRA? What if that is not their @apache address? On Mon, Aug 25, 2014 at 9:56 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for your input, Steve. Sorry for sending the email out that late, I sent it as soon as I could. On Mon, Aug 25, 2014 at 2:20 AM, Steve Loughran ste...@hortonworks.com wrote: just caught up with this after some offlininess...15:48 PST is too late for me. I'd be -1 to a change to master because of that risk that it does break existing code -especially people that have trunk off the git mirrors and automated builds/merges to go with it. Fair enough. It makes sense to leave it as trunk, unless someone is against it being trunk. master may be viewed as the official git way, but it doesn't have to be. For git-flow workflows (which we use in slider) master/ is for releases, develop/ for dev. On 24 August 2014 02:31, Karthik Kambatla ka...@cloudera.com wrote: Couple of things: 1. Since no one expressed any reservations against doing this on Sunday or renaming trunk to master, I ll go ahead and confirm that. I think that serves us better in the long run. 2. Arpit brought up the precommit builds - we should definitely fix them as soon as we can. I understand Giri maintains those builds, do we have anyone else who has access in case Giri is not reachable? Giri - please shout out if you can help us with this either on Sunday or Monday. Thanks Karthik On Fri, Aug 22, 2014 at 3:50 PM, Karthik Kambatla ka...@cloudera.com wrote: Also, does anyone know what we use for integration between JIRA and svn? I am assuming svn2jira. On Fri, Aug 22, 2014 at 3:48 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, For the SCM migration, feel free to follow https://issues.apache.org/jira/browse/INFRA-8195 Most of this is planned to be handled this Sunday. As a result, the subversion repository would be read-only. If this is a major issue for you, please shout out. Daniel Gruno, the one helping us with the migration, was asking if we are open to renaming trunk to master to better conform to git lingo. I am tempted to say yes, but wanted to check. Would greatly appreciate any help with checking the git repo has everything. Thanks Karthik -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message
Re: hadoop-2.5 - June end?
We should also include extended attributes feature for HDFS from HDFS-2006 for release 2.5. On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create an intermediate release (2.5) this month - I think ATS security is pretty close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of the month, with a branch a couple of weeks prior. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: #Contributors on JIRA
Last time we cleaned up names of people who had not contributed in a long time. That could be an option. On Mon, May 12, 2014 at 12:03 PM, Karthik Kambatla ka...@cloudera.comwrote: Hi devs Looks like we ran over the max contributors allowed for a project, again. I don't remember what we did last time and can't find it in my email either. Can we bump up the number of contributors allowed? Otherwise, we might have to remove some of the currently inactive contributors from the list? Thanks Karthik -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.4.0
+1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. I agree with Arun's anaylysis. At this time, the bar for blockers should be quite high. We can do a dot release if people want some more bug fixes. On Mon, Mar 31, 2014 at 2:22 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.4.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10311) Cleanup vendor names in the code base
Suresh Srinivas created HADOOP-10311: Summary: Cleanup vendor names in the code base Key: HADOOP-10311 URL: https://issues.apache.org/jira/browse/HADOOP-10311 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Suresh Srinivas Priority: Blocker -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HADOOP-10292) Restore HttpServer from branch-2.2 in branch-2
[ https://issues.apache.org/jira/browse/HADOOP-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-10292. -- Resolution: Fixed Hadoop Flags: Reviewed I have committed this change to branch-2. Thank you [~wheat9]. Thank you [~stack] for testing and review. HttpServer needs to be removed in branch-2 once HBase stops using it from Hadoop Common. Restore HttpServer from branch-2.2 in branch-2 -- Key: HADOOP-10292 URL: https://issues.apache.org/jira/browse/HADOOP-10292 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.3.0 Attachments: HADOOP-10292.000.patch This jira is a follow-up jira of HADOOP-10255. It brings in the HttpServer in branch-2.2 directly into branch-2 to restore the compatibility of HBase. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: Logistics for releasing 2.4
There is not much progress on symlinks issue. I think we should move forward with 2.4 release with symlinks disabled. Status of 2.4 features from HDFS so far: - HDFS-2832 Heterogeneous storage support has been merged - HDFS-5535 rolling upgrades work is in progress - HDFS-4685 ACL related work is close to completion - HDFS-4949 As Andrew has proposed, this will be soon merged into 2.4 Regards, Suresh On Tue, Jan 21, 2014 at 11:12 AM, Arun C Murthy a...@hortonworks.com wrote: Andrew, I'm almost ready to push out rc0 for 2.3 (been testing it overnight), I'm pretty sure I'll get that out tonight. However, AHS (YARN-321) is very close (merge vote going on) … so that will definitely make it in very soon. So, my plan is essentially the same i.e. release 2.4 end of the month (after a bit more testing of RM HA in secure mode). Thanks for the offer, I'll ping you if I need any help. OTOH, can someone from HDFS chime in on status of symlinks? Arun On Jan 20, 2014, at 4:19 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I'm pretty excited to see a 2.4 this month if possible. Since I think people were favorable to the idea of time-based releases, how do we feel about just cutting branch-2 and spinning up the release process for our January goal? Looking at the roadmap (https://wiki.apache.org/hadoop/Roadmap), on the HDFS side, I plan to post a branch-2 patch for HDFS-4949 this week, and HDFS-2832 is already in. On the YARN side, it appears that RM HA is in, but the other three features (AHS, unmanaged containers, and dynamic resource configuration) remain unresolved. I think a 2.4 with HDFS-4949, HDFS-2832, and YARN-149 is already a pretty nice release. If it'd help, I'm willing to volunteer as release manager to help get this out the door. Thoughts welcome! Thanks, Andrew -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk
Great work Arpit and Nicholas! +1. I have been part of design. I have been following the changes closely. This is ready to be merged into trunk. On Mon, Dec 2, 2013 at 4:06 PM, Arpit Agarwal aagar...@hortonworks.comwrote: Hello all, I would like to call a vote to merge phase 1 of the Heterogeneous Storage feature into trunk. *Scope of the changes:* The changes allow exposing the DataNode as a collection of storages and set the foundation for subsequent work to present Heterogeneous Storages to applications. This allows DataNodes to send block and storage reports per-storage. In addition this change introduces the ability to add a 'storage type' tag to the storage directories. This enables supporting different types of storages in addition to disk storage. Development of the feature is tracked in the jira https://issues.apache.org/jira/browse/HDFS-2832. *Details of development and testing:* Development has been done in a separate branch - https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The updated design is posted at - https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf . The changes involve ~6K changed lines of code, with a third of those changes being to tests. Please see the test plan https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor the details. Once the feature is merged into trunk, we will continue to test and fix any bugs that may be found on trunk as well as add further tests as outlined in the test plan. The bulk of the design and implementation was done by Suresh Srinivas, Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and Todd Lipcon for providing feedback on the Jiras and in discussions. This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT. Thanks, Arpit -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] What is the purpose of merge vote threads?
I agree with what Nicholas is saying. Feature branch merge votes are similar to traditional review-commit process. That means the code should be ready, and pass the Jenkins build tests. Also similar to regular patches where one describes what changes the patch brings, having an updated design document (with change history for the updates made to the design) helps people understand the design. Updated user document for the feature helps end users to understand how the feature works and helps them participate in testing. Finally, as expected by every patch, we should have unit tests and also details of manual tests done (with test plan for a feature). This helps people participate in the voting/review more effectively. There can be exceptions to the above list and some of the work could be deferred. That could be captured in the jira, with the plan of how that work gets done later. In such cases in the past we had deferred merging the feature from trunk to a release branch until the work was completed in trunk. Granted some of the feature readiness activity can be done during voting. But I fail to understand why expediting a feature that takes months to build should try to optimize a week. Why not finish the requirements we have for every patch for feature branch also? On Thu, Oct 24, 2013 at 6:19 PM, Tsz Wo (Nicholas), Sze s29752-hadoop...@yahoo.com wrote: (Resend) No. In the past, committers would merge a branch once the merge vote had been passed even there were problems in the branch. Below is my understanding of merge vote. Feature branch merge votes is the same as the traditional code review-commit process except that it requires three +1's and it happens in the mailing list. For review-commit, we +1 on the patch. If we think that the patch needs to be changed, we should ask the contributor posting a new patch before +1. This is not strictly enforced. In some cases, committers may say something like +1 once X and Y have been fixed. In some worse cases, a committer may has committed a patch without +1 and then his friend committer will say I mean +1 by my previous comment but I forgot to post it. Sorry, here is my +1. For merge vote, it should be considered that a big patch is generated by the diff from the branch over trunk. Then, committers vote on the big patch in the mailing list. As review-commit, if the patch need to be changed, committers should not +1 on it. Unfortunately, it is generally hard to review big patches and it is relatively easy to sneak bad code in. Both review-commit and merge vote are similar to voting on release candidates -- we vote on the bits of the candidate but neither vote on an idea nor a plan. (Of course, there are other types of vote for voting on a plan.) Regards, Tsz-Wo On Thursday, October 24, 2013 5:09 PM, Tsz Wo Sze szets...@yahoo.com wrote: No. In the past, committers would merge a branch once the merge vote had been passed even there were problems in the branch. Below is my understanding of merge vote. Feature branch merge votes is the same as the traditional code review-commit process except that it requires three +1's and it happens in the mailing list. For review-commit, we +1 on the patch. If we think that the patch needs to be changed, we should ask the contributor posting a new patch before +1. This is not strictly enforced. In some cases, committers may say something like +1 once X and Y have been fixed. In some worse cases, a committer may has committed a patch without +1 and then his friend committer will say I mean +1 by my previous comment but I forgot to post it. Sorry, here is my +1. For merge vote, it should be considered that a big patch is generated by the diff from the branch over trunk. Then, committers vote on the big patch in the mailing list. As review-commit, if the patch need to be changed, committers should not +1 on it. Unfortunately, it is generally hard to review big patches and it is relatively easy to sneak bad code in. Both review-commit and merge vote are similar to voting on release candidates -- we vote on the bits of the candidate but neither vote on an idea nor a plan. (Of course, there are other types of vote for voting on a plan.) Regards, Tsz-Wo On Thursday, October 24, 2013 3:46 PM, Doug Cutting cutt...@apache.org wrote: On Thu, Oct 24, 2013 at 2:51 PM, Chris Nauroth cnaur...@hortonworks.com wrote: When the voting happens on jira with a finalized merge patch, I know exactly what I'm voting for, because it's the same review-and-commit process that we follow every day with the extra requirement of 3 +1s. When it happens on email, it's less clear to me exactly what I'm voting for. Here's my take, FWIW. The entire project needs to determine whether it is willing to take on the maintenance of code developed in a branch. This vote needs
Re: [VOTE] Merge HDFS-4949 to trunk
I posted a comment in the other thread about feature branch merges. My preference is to make sure the requirements we have for regular patches be applied to feature branch patch as well (3 +1s is the only exception). Also adding details about what functionality is missing (I posted a comment on HDFS-4949) and the changes that deferred that will be done post merge to trunk would be good. It would be better to start the merge vote when the work is ready instead of trying to optimize 1 week by doing the required work for merging in parallel with the vote. If all the requirements for merging have been met, I am +1 on the merge, without the need for restarting the vote. On Thu, Oct 24, 2013 at 11:29 PM, Aaron T. Myers a...@apache.org wrote: I don't necessarily disagree with the general questions about the procedural issues of merge votes. Thanks for bringing that up in the other thread you mentioned. To some extent it seems like much of this has been based on custom, and if folks feel that more precisely defining the merge vote process is warranted, then I think we should take that up over on that thread. With regard to this particular merge vote, I've spoken with Chris offline about his feelings on this. He said that he is not dead-set on restarting the vote, though he suspects that others may be. It seems to me the remaining unfinished asks (e.g. updating the design doc) can reasonably be done either after this vote but before the merge to trunk proper, or could even reasonably be done after merging to trunk. Given that, I'll lend my +1 to this merge. I've been reviewing the branch pretty consistently since work started on it, and have personally run/tested several builds of it along the way. I've also reviewed the design thoroughly. The implementation, overall design, and API seem to me plenty stable enough to be merged into trunk. I know that there remains a handful of javac warnings in the branch that aren't in trunk, but I trust those will be taken care of before the merge. If anyone out there does feel strongly that this merge vote should be restarted, then please speak up soon. Again, we can restart the vote if need be, but I honestly think we'll gain very little by doing so. Best, Aaron On Fri, Oct 25, 2013 at 5:45 AM, Chris Nauroth cnaur...@hortonworks.com wrote: Hi Andrew, I've come to the conclusion that I'm very confused about merge votes. :-) It's not just about HDFS-4949. I'm confused about all merge votes. Rather than muddy the waters here, I've started a separate discussion on common-dev. I do agree with the general plan outlined here, and I will comment directly on the HDFS-4949 jira with a binding +1 when I see that we've completed that plan. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, Oct 23, 2013 at 2:18 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Chris, Right now we're on track to have all of those things done by tomorrow. Since the remaining issues are either not technical or do not involve major changes, I was hoping we could +1 this merge vote in the spirit of +1 pending jenkins. We've gotten clean unit test runs on upstream Jenkins as well, so the only fixups we should need for test-patch.sh are findbugs and javac (which are normally pretty trivial to clean up). Of course, all of your listed prereqs and test-patch would be taken care of before actually merging to trunk. So, we can reset the vote if you feel strongly about this, but it seems like the only real result will be delaying the merge by a week. Thanks, Andrew On Wed, Oct 23, 2013 at 1:03 PM, Chris Nauroth cnaur...@hortonworks.com wrote: I've received some feedback that we haven't handled this merge vote the same as other comparable merge votes, and that the vote should be reset because of this. The recent custom is that we only call for the merge vote after all pre-requisites have been satisfied. This would include committing to the feature branch all patches that the devs deem necessary before the code lands in trunk, posting a test plan, posting an updated design doc in case implementation choices diverged from the original design doc, and getting a good test-patch run from Jenkins on the merge patch. This was the process followed for other recent major features like HDFS-2802 (snapshots), HDFS-347 (short-circuit reads via sharing file descriptors), and HADOOP-8562 (Windows compatibility). In this thread, we've diverged from that process by calling for a vote on a branch that hasn't yet completed the pre-requisites and stating a plan for work to be done before the merge. I still support this work, but can we please restart the vote after the pre-requisites have landed in the branch? Chris Nauroth Hortonworks http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (binding) On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HADOOP-10042) Heap space error during copy from maptask to reduce task
[ https://issues.apache.org/jira/browse/HADOOP-10042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-10042. -- Resolution: Invalid Heap space error during copy from maptask to reduce task Key: HADOOP-10042 URL: https://issues.apache.org/jira/browse/HADOOP-10042 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 1.2.1 Environment: Ubuntu cluster Reporter: Dieter De Witte Fix For: 1.2.1 Attachments: mapred-site.OLDxml http://stackoverflow.com/questions/19298357/out-of-memory-error-in-mapreduce-shuffle-phase I've described the problem on stackoverflow as well. It contains a link to another JIRA: http://hadoop-common.472056.n3.nabble.com/Shuffle-In-Memory-OutOfMemoryError-td433197.html My errors are completely the same: out of memory error when mapred.job.shuffle.input.buffer.percent = 0.7, the program does work when I put it to 0.2, does this mean the original JIRA was not resolved? Does anybody have an idea whether this is a mapreduce issue or is it a misconfiguration from my part? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10035) Cleanup TestFilterFileSystem
Suresh Srinivas created HADOOP-10035: Summary: Cleanup TestFilterFileSystem Key: HADOOP-10035 URL: https://issues.apache.org/jira/browse/HADOOP-10035 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.1.1-beta Reporter: Suresh Srinivas Currently TestFilterFileSystem only checks for FileSystem methods that must be implemented in FilterFileSystem with a list of methods that are exception to this rule. This jira wants to make this check stricter by adding a test for ensuring the methods in exception rule list must not be implemented by the FilterFileSystem. This also cleans up the current class that has methods from exception rule list to interface to avoid having to provide dummy implementation of the methods. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HADOOP-10020) disable symlinks temporarily
[ https://issues.apache.org/jira/browse/HADOOP-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-10020. -- Resolution: Fixed Fix Version/s: 2.1.2-beta Release Note: During review of symbolic links, many issues were found related impact on semantics of existing APIs such FileSystem#listStatus, FileSystem#globStatus etc. There were also many issues brought up about symbolic links and the impact on security and functionality of HDFS. All these issues will be address in the upcoming release 2.3. Until then the feature is temporarily disabled. I have committed this change to 2.1-beta branch. disable symlinks temporarily Key: HADOOP-10020 URL: https://issues.apache.org/jira/browse/HADOOP-10020 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.2-beta Reporter: Colin Patrick McCabe Assignee: Sanjay Radia Priority: Blocker Fix For: 2.1.2-beta Attachments: Hadoop-10020-2.patch, Hadoop-10020-3.patch, Hadoop-10020-4-forBranch2.1beta.patch, Hadoop-10020-4.patch, Hadoop-10020.patch disable symlinks temporarily until we can make them production-ready in Hadoop 2.3 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10022) Add support for per project https support
Suresh Srinivas created HADOOP-10022: Summary: Add support for per project https support Key: HADOOP-10022 URL: https://issues.apache.org/jira/browse/HADOOP-10022 Project: Hadoop Common Issue Type: Bug Reporter: Suresh Srinivas Current configuration hadoop.https.enable turns on https only support for all the daemons in hadoop. This is an umbrella jira to add per project https configuration. For more details, see the detailed proposal - https://issues.apache.org/jira/browse/HADOOP-8581?focusedCommentId=13784332page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13784332 The current scope of work is descriged in - https://issues.apache.org/jira/browse/HADOOP-8581?focusedCommentId=13786567page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13786567 a -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10023) Add https support in HDFS
Suresh Srinivas created HADOOP-10023: Summary: Add https support in HDFS Key: HADOOP-10023 URL: https://issues.apache.org/jira/browse/HADOOP-10023 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas This is the HDFS part of HADOOP-10022. This will serve as the umbrella jira for all the https related cleanup in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)
(This time copying all the lists) I am +1 for naming the new branch 2.2.0. On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy a...@hortonworks.com wrote: Guys, I took a look at the content in 2.1.2-beta so far, other than the critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR, there is fairly little content (unit tests fixes etc.) Furthermore, it's standing up well in testing too. Plus, the protocols look good for now (I wrote a gohadoop to try convince myself), let's lock them in. Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than make another 2.1.x release. This will drop a short-lived release (2.1.2) and help us move forward on 2.3 which has a fair bunch of content already... Thoughts? thanks, Arun On Sep 24, 2013, at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote: I've added MAPREDUCE-5531 to the blocker list. - Zhijie On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com wrote: With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
+1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: symlink support in Hadoop 2 GA
I agree that this is an important change. However, 2.2.0 GA is getting ready to rollout in weeks. I am concerned that these changes will add not only incompatible changes late in the game, but also possibly instability. Java API incompatibility is some thing we have avoided for the most part and I am concerned that this is adding such incompatibility in FileSystem APIs. We should find work arounds by adding possibly newer APIs and leaving existing APIs as is. If this can be done, my vote is to enable this feature in 2.3. Even if it cannot be done, I am concerned that this is coming quite late and we should see if could allow some incompatible changes into 2.3 for this feature. On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang andrew.w...@cloudera.comwrote: Hi all, I wanted to broadcast plans for putting the FileSystem symlinks work (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think it's pretty important we get it in since it's not a compatible change; if it misses the GA train, we're not going to have symlinks until the next major release. However, we're still dealing with ongoing issues revealed via testing. There's user-code out there that only handles files and directories and will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912 for a nice example where globStatus returning symlinks broke Pig; some of us had a conference call to talk it through, and one definite conclusion was that this wasn't solvable in a generally compatible manner. There are also still some gaps in symlink support right now. For example, the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink resolution, and tooling like the FsShell and Distcp still need to be updated as well. So, there's definitely work to be done, but there are a lot of users interested in the feature, and symlinks really should be in GA. Would appreciate any thoughts/input on the matter. Thanks, Andrew -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Hadoop SSO/Token Server Components
One aside: if you come across a bug, please try to fix it upstream and then merge into the feature branch rather than cherry-picking patches or only fixing it on the branch. It becomes very awkward to track. -C Related to this, when refactoring the code, generally required for large feature development, consider first refactoring in trunk and then make additional changes for the feature in the feature branch. This helps a lot in being able to merge the trunk to feature branch periodically. This will also help in keeping the change for merging feature to trunk small and easier reviews. Regards, Suresh -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Reopened] (HADOOP-9745) TestZKFailoverController test fails
[ https://issues.apache.org/jira/browse/HADOOP-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HADOOP-9745: - TestZKFailoverController test fails --- Key: HADOOP-9745 URL: https://issues.apache.org/jira/browse/HADOOP-9745 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Jean-Baptiste Onofré - testGracefulFailoverFailBecomingActive(org.apache.hadoop.ha.TestZKFailoverController): Did not fail to graceful failover when target failed to become active! - testGracefulFailoverFailBecomingStandby(org.apache.hadoop.ha.TestZKFailoverController): expected:1 but was:0 - testGracefulFailoverFailBecomingStandbyAndFailFence(org.apache.hadoop.ha.TestZKFailoverController): Failover should have failed when old node wont fence -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9745) TestZKFailoverController test fails
[ https://issues.apache.org/jira/browse/HADOOP-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9745. - Resolution: Not A Problem TestZKFailoverController test fails --- Key: HADOOP-9745 URL: https://issues.apache.org/jira/browse/HADOOP-9745 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Jean-Baptiste Onofré - testGracefulFailoverFailBecomingActive(org.apache.hadoop.ha.TestZKFailoverController): Did not fail to graceful failover when target failed to become active! - testGracefulFailoverFailBecomingStandby(org.apache.hadoop.ha.TestZKFailoverController): expected:1 but was:0 - testGracefulFailoverFailBecomingStandbyAndFailFence(org.apache.hadoop.ha.TestZKFailoverController): Failover should have failed when old node wont fence -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9381) Document dfs cp -f option
[ https://issues.apache.org/jira/browse/HADOOP-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HADOOP-9381: - Notices that the document still does not capture -f option, though the copy command does. Document dfs cp -f option - Key: HADOOP-9381 URL: https://issues.apache.org/jira/browse/HADOOP-9381 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Keegan Witt Priority: Trivial Attachments: HADOOP-9381.patch dfs cp should document -f (overwrite) option in the page displayed by -help. Additionally, the HTML documentation page should also document this option and all the options should all be formatted the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (HADOOP-9822) create constant maxCapacity in RetryCache rather than hard-coding 16 in RetryCache constructor
I do not think this is worth making a change. Even Java code has these kinds of instances. This is essentially private static used in one single class, only in the constructor. That said, if you still want to make this change, it is not maxCapacity. It is DEFAULT_INITIAL_CAPACITY. On Aug 1, 2013, at 10:11 PM, Tsuyoshi OZAWA (JIRA) j...@apache.org wrote: Tsuyoshi OZAWA created HADOOP-9822: -- Summary: create constant maxCapacity in RetryCache rather than hard-coding 16 in RetryCache constructor Key: HADOOP-9822 URL: https://issues.apache.org/jira/browse/HADOOP-9822 Project: Hadoop Common Issue Type: Bug Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor The magic number 16 is also used in ClientId.BYTE_LENGTH, so hard-coding magic number 16 is a bit confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9793) RetryInvocationHandler uses raw types that should be parameterized
Suresh Srinivas created HADOOP-9793:
---
Summary: RetryInvocationHandler uses raw types that should be
parameterized
Key: HADOOP-9793
URL: https://issues.apache.org/jira/browse/HADOOP-9793
Project: Hadoop Common
Issue Type: Bug
Reporter: Suresh Srinivas
Priority: Minor
This causes javac warnings as shown below:
{noformat}
274c274,275
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java:[147,45]
[unchecked] unchecked call to performFailover(T) as a member of the raw type
org.apache.hadoop.io.retry.FailoverProxyProvider
---
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java:[103,24]
[unchecked] unchecked call to
getMethod(java.lang.String,java.lang.Class?...) as a member of the raw type
java.lang.Class
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java:[152,45]
[unchecked] unchecked call to performFailover(T) as a member of the raw type
org.apache.hadoop.io.retry.FailoverProxyProvider
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9792) Retry the methods that are tagged @AtMostOnce along with @Idempotent
Suresh Srinivas created HADOOP-9792: --- Summary: Retry the methods that are tagged @AtMostOnce along with @Idempotent Key: HADOOP-9792 URL: https://issues.apache.org/jira/browse/HADOOP-9792 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HADOOP-9792.patch Currently the operations marked as @Idempotent are retried. Now that we have added new operations that are marked @AtMostOnce that guaranteed to be executed only once with the help of RetryCache on server implementation, these methods should also be retried. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9770) RetryCache#state need not be volatile
Suresh Srinivas created HADOOP-9770: --- Summary: RetryCache#state need not be volatile Key: HADOOP-9770 URL: https://issues.apache.org/jira/browse/HADOOP-9770 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.1.0-beta Reporter: Suresh Srinivas Priority: Minor See the comment - https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9764) MapReduce output issue
[
https://issues.apache.org/jira/browse/HADOOP-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9764.
-
Resolution: Invalid
MapReduce output issue
--
Key: HADOOP-9764
URL: https://issues.apache.org/jira/browse/HADOOP-9764
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.0.3
Environment: ubuntu
Reporter: Mullangi
Hi,
I am new to Hadoop concepts.
While practicing with one custom MapReduce program, I found the result is not
as expected after executing the code on HDFS based file. Please note that
when I execute the same program using Unix based file,getting expected result.
Below are the details of my code.
MapReduce in java
==
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.util.*;
public class WordCount1 {
public static class Map extends MapReduceBase implements Mapper {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector output,
Reporter reporter) throws IOException {
String line = value.toString();
String tokenedZone=null;
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
tokenedZone=tokenizer.nextToken();
word.set(tokenedZone);
output.collect(word, one);
}
}
}
public static class Reduce extends MapReduceBase implements Reducer {
public void reduce(Text key, Iterator values, OutputCollector output,
Reporter reporter) throws IOException {
int sum = 0;
int val = 0;
while (values.hasNext()) {
val = values.next().get();
sum += val;
}
if(sumgt;1)
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf();
conf.setJarByClass(WordCount1.class);
conf.setJobName(wordcount1);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
Path inPath = new Path(args[0]);
Path outPath = new Path(args[0]);
FileInputFormat.setInputPaths(conf,inPath );
FileOutputFormat.setOutputPath(conf, outPath);
JobClient.runJob(conf);
}
}
input File
===
test my program
during test and my hadoop
your during
get program
hadoop generated output file on HDFS file system
===
during2
my2
test 2
hadoop generated output file on local file system
===
during2
my2
program 2
test 2
Please help me on this issue
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9760) Move GSet and LightWeightGSet to hadoop-common
Suresh Srinivas created HADOOP-9760: --- Summary: Move GSet and LightWeightGSet to hadoop-common Key: HADOOP-9760 URL: https://issues.apache.org/jira/browse/HADOOP-9760 Project: Hadoop Common Issue Type: Bug Components: util Reporter: Suresh Srinivas Assignee: Suresh Srinivas GSet and related classes are useful for all of Hadoop. This jira proposes moving it to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9762) RetryCache utility for implementing RPC retries
Suresh Srinivas created HADOOP-9762: --- Summary: RetryCache utility for implementing RPC retries Key: HADOOP-9762 URL: https://issues.apache.org/jira/browse/HADOOP-9762 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Suresh Srinivas HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira moves this utility to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8873) Port HADOOP-8175 (Add mkdir -p flag) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8873. - Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I committed the patch to branch-1. Thank you [~ajisakaa]. Port HADOOP-8175 (Add mkdir -p flag) to branch-1 Key: HADOOP-8873 URL: https://issues.apache.org/jira/browse/HADOOP-8873 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.0 Reporter: Eli Collins Assignee: Akira AJISAKA Labels: newbie Fix For: 1.3.0 Attachments: HADOOP-8873-1.patch, HADOOP-8873-2.patch, HADOOP-8873-3.patch, HADOOP-8873-branch-1-4.patch Per HADOOP-8551 let's port the mkdir -p option to branch-1 for a 1.x release to help users transition to the new shell behavior. In Hadoop 2.x mkdir currently requires the -p option to create parent directories but a program that specifies it won't work on 1.x since it doesn't support this option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9717) Add retry flag/retry attempt count to the RPC requests
Suresh Srinivas created HADOOP-9717: --- Summary: Add retry flag/retry attempt count to the RPC requests Key: HADOOP-9717 URL: https://issues.apache.org/jira/browse/HADOOP-9717 Project: Hadoop Common Issue Type: Improvement Reporter: Suresh Srinivas RetryCache lookup on server side implementation can be optimized if Rpc request indicates if the request is being retried. This jira proposes adding an optional field to Rpc request that indicates if request is being retried. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: protobuf 2.5.0 failure should be known by wiki
Now I know HADOOP-9346 and HADOOP-9440 for enabling protobuf 2.5.0, but these issues have been left for months. Have you seen the reason why these issues have not been fixed; see - https://issues.apache.org/jira/browse/HADOOP-9346?focusedCommentId=13657835page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13657835 If you have a solution for that issue, I am glad to commit that patch. I tried to edit wiki but it seems that I don't have the permission. How can I edit? Steve already answered this. Regards, Suresh -- http://hortonworks.com/download/
[jira] [Created] (HADOOP-9702) FileSystem should fail create call with OverWrite and Append flags
Suresh Srinivas created HADOOP-9702: --- Summary: FileSystem should fail create call with OverWrite and Append flags Key: HADOOP-9702 URL: https://issues.apache.org/jira/browse/HADOOP-9702 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Suresh Srinivas OverWrite flag means delete the existing file. Append flag means append to an existing file. Both cannot be enabled at the same time. Following combinations are okay: # Create Overwrite is okay - if file does not exist, it is created. If it exists, it will be overwritten. # Create Append is okay - if file does not exist, create it. If the file exists append to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9688) Add globally unique request ID to RPC requests
Suresh Srinivas created HADOOP-9688: --- Summary: Add globally unique request ID to RPC requests Key: HADOOP-9688 URL: https://issues.apache.org/jira/browse/HADOOP-9688 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Suresh Srinivas Assignee: Suresh Srinivas This is a subtask in hadoop-common related to HDFS-4942 to add unique request ID to RPC requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: creating 2.2.0 version in JIRA
I have added in to HDFS, HADOOP, MAPREDUCE projects. Can someone add it for YARN? On Fri, Jun 21, 2013 at 9:35 AM, Alejandro Abdelnur t...@cloudera.comwrote: When Arun created branch-2.1-beta he stated: The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet). I working/committing some JIRAs that I'm putting in branch-2 (testcases and improvements) but I don't want to put them in branch-2.1-beta as they are not critical and I don't won't add unnecessary noise to the branch-2.1-beta release work. Currently branch-2 POMs have a version 2.2.0 and the CHANGES.txt files as well. But because we did not create a JIRA version I cannot close those JIRAs. Can we please create the JIRA versions? later we can rename them. Thx -- Alejandro -- http://hortonworks.com/download/
[jira] [Resolved] (HADOOP-9615) Hadoop Jar command not working when used with Spring ORM
[ https://issues.apache.org/jira/browse/HADOOP-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9615. - Resolution: Invalid I agree that this is not a Hadoop issue. I am going to close this as Invalid. Please re-open if you disagree with reasons why this is a Hadoop issue. Hadoop Jar command not working when used with Spring ORM Key: HADOOP-9615 URL: https://issues.apache.org/jira/browse/HADOOP-9615 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.0-alpha Environment: CentOS, Reporter: Deepa Vasanthkumar Labels: hadoop-2.0 Unable to invoke 'hadoop jar' command for class, which contains Spring persistance unit. The problem is that, the jar file uses Spring ORM for loading the persistance configurations, and based on these configurations, i need to move the files to HDFS. While invoking the jar with hadoop jar command (having spring orm injected) the exception is as: Exception in thread main org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor#0' defined in class path resource [applicationContext.xml Error creating bean with name 'entityManagerFactory' defined in class path resource [applicationContext.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Conflicting persistence unit definitions for name 'Persistance': file:/home/user/Desktop/ABC/apnJar.jar, file:/tmp/hadoop-user/hadoop-unjar2841422106164401019/ Caused by: java.lang.IllegalStateException: Conflicting persistence unit definitions for name 'Persistance': -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8957) AbstractFileSystem#IsValidName should be overridden for embedded file systems like ViewFs
[ https://issues.apache.org/jira/browse/HADOOP-8957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8957. - Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed I merged the change to branch-2. AbstractFileSystem#IsValidName should be overridden for embedded file systems like ViewFs - Key: HADOOP-8957 URL: https://issues.apache.org/jira/browse/HADOOP-8957 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 2.1.0-beta Attachments: HADOOP-8957-branch-2.patch, HADOOP-8957-branch-trunk-win.2.patch, HADOOP-8957-branch-trunk-win.3.patch, HADOOP-8957-branch-trunk-win.4.patch, HADOOP-8957.patch, HADOOP-8957.patch, HADOOP-8957-trunk.4.patch This appears to be a problem with parsing a Windows-specific path, ultimately throwing InvocationTargetException from AbstractFileSystem.newInstance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9131) TestLocalFileSystem#testListStatusWithColons cannot run on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9131.
-
Resolution: Fixed
Fix Version/s: (was: 3.0.0)
2.1.0-beta
I have merged the change into branch-2. Thank you Chuan for bringing this up.
TestLocalFileSystem#testListStatusWithColons cannot run on Windows
--
Key: HADOOP-9131
URL: https://issues.apache.org/jira/browse/HADOOP-9131
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.1.0-beta
Attachments: HADOOP-9131.1.patch
HADOOP-8962 added a new test,
{{TestLocalFileSystem#testListStatusWithColons}}, covering the case of files
that contain ':'. This test cannot pass on Windows, because on Windows, the
local file system does not support ':' in file names.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8958. - Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.1.0-beta I committed the patch to branch-2. ViewFs:Non absolute mount name failures when running multiple tests on Windows -- Key: HADOOP-8958 URL: https://issues.apache.org/jira/browse/HADOOP-8958 Project: Hadoop Common Issue Type: Bug Components: viewfs Affects Versions: 3.0.0, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.1.0-beta Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: HADOOP-8545 OpenStack Swift patch ready for commit -reviewers needed
Steve, I will review this. Might need a couple of days though. On Mon, Jun 3, 2013 at 7:12 AM, Steve Loughran ste...@hortonworks.comwrote: Hi, We've got the HADOOP-8545 https://issues.apache.org/jira/browse/HADOOP-8545patch stable enough that I'd like it checked in and in the 2.1 beta so that we can see what problems occur in the field. The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't break anything in hadoop-common/hdfs/yarn, and it includes more method-by-method tests than any of the non-HDFS filesystem clients have, including basic scale tests and the corner-case problems that we're going to have to talk about in terms of FS API spec and tests. In github, there are some very basic scale tests running Pig jobs against a configurable amount of data, showing that we can run pig against GB of data. We've run these tests against rackspace US, rackspace US and hpcloud -the public Swift services -as well as private deployments of the latest version of OpenStack. This makes us confident that we've got authentication right, and as we get consistent outcomes from all clusters, we aren't relying some cluster-/provider- specific detail. So: please can people review it! The site documentation (did I mention we've got that?) lists how to connect to a swift service, and how to test against them. If anyone does want to run the test themselves, try the docs and then escalate to us if there are any problems. Steve Loughran, Dmitry Mezhenskiy (Mirantis) David Dobbins (Rackspace) -- http://hortonworks.com/download/
[jira] [Reopened] (HADOOP-8562) Enhancements to support Hadoop on Windows Server and Windows Azure environments
[ https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HADOOP-8562: - Reopening the issue for merging. Enhancements to support Hadoop on Windows Server and Windows Azure environments --- Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 3.0.0 Attachments: branch-2.merge.patch, branch-2.merge.patch, branch-trunk-win.min-notest.patch, branch-trunk-win-min.patch, branch-trunk-win.min.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, test-untar.tar, test-untar.tgz This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8562) Enhancements to support Hadoop on Windows Server and Windows Azure environments
[ https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8562. - Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.0.5-beta I merge this patch into branch-2. Thank you every one for testing. Can you folks run another quick test to verify things are merged correctly? Enhancements to support Hadoop on Windows Server and Windows Azure environments --- Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.5-beta Attachments: branch-2.merge.patch, branch-2.merge.patch, branch-trunk-win.min-notest.patch, branch-trunk-win-min.patch, branch-trunk-win.min.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, test-untar.tar, test-untar.tgz This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [PROPOSAL] change in bylaws to remove Release Plan vote
+1 On Tue, May 21, 2013 at 2:10 PM, Matt Foley ma...@apache.org wrote: Hi all, This has been a side topic in several email threads recently. Currently we have an ambiguity. We have a tradition in the dev community that any committer can create a branch, and propose release candidates from it. Yet the Hadoop bylaws say that releases have to be planned in advance, the plan needs to be voted on, and presumably can be denied. Apache policies (primarily here http://www.apache.org/dev/release.html and here http://www.apache.org/foundation/voting.html, with non-normative commentary here http://incubator.apache.org/guides/releasemanagement.html#best-practice) are very clear on how Releases have to be approved, and our bylaws are consistent with those policies. But Apache policies don't say anything I've found about Release Plans, nor about voting on Release Plans. I propose the following change, to remove Release Plan votes, and give a simple definition of Release Manager role. I'm opening discussion with this proposal, and will put it to a vote if we seem to be getting consensus. Here's the changes I suggest in the Bylawshttp://hadoop.apache.org/bylaws.html document: === 1. In the Decision Making : Actions section of the Bylaws, the following text is removed: ** Release Plan* Defines the timetable and actions for a release. The plan also nominates a Release Manager. Lazy majority of active committers 2. In the Roles and Responsibilities section of the Bylaws, an additional role is defined: ** Release Manager* A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToReleasehttps://wiki.apache.org/hadoop/HowToRelease. The RM shall publish a Release Plan on the *common-dev@* list stating the branch from which they intend to make a Release Candidate, at least one week before they do so. The RM is responsible for building consensus around the content of the Release Candidate, in order to achieve a successful Product Release vote. === Please share your views. Best regards, --Matt (long-time release manager) -- http://hortonworks.com/download/
[jira] [Resolved] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9523. - Resolution: Fixed Marking the issue as resolved. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] - Release 2.0.5-beta
This is the course that we were taking before the unfortunate disruption. We should be able to meet both the stabilization goals and compatibility goals quickly with this proposal. I personally am willing to invest a lot of time in testing, code reviews and work on adding missing functionality to ensure the goal of this proposal is successful. +1. On Wed, May 15, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, A considerable number of people have expressed confusion regarding the recent vote on 2.0.5, beta status etc. given lack of specifics, the voting itself (validity of the vote itself, whose votes are binding) etc. IMHO technical arguments (incompatibility b/w 2.0 2.1, current stability of 3 features under debate etc.) have been lost in the discussion in favor of non-technical (almost dramatic) nuances such as seizing the moment. There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1) - this is a red flag for me; particularly when there are just 3 features being debated and active committers and contributors are confident of and ready to stand by their work. All patches, I believe, are ready to be merged in the the next few days per discussions on jira. This will, clearly, not delay the other API work which everyone agrees is crucial. As a result, I feel no recourse but to restart a new vote - all attempts at calm, reasoned, civil discussion based on technical arguments have come to naught - I apologize for the thrash caused to everyone's attention. To get past all of this confusion, I'd like to present an alternate, specific proposal for consideration. I propose we continue the original plan and make a 2.0.5-beta release by May end with the following content: # HDFS-347 # HDFS Snapshots # Windows support # Necessary final API/protocol changes such as: * Final YARN API changes: YARN-386 * MR Binary Compatibility: MAPREDUCE-5108 * Final RPC cleanup: HADOOP-8990 People working on the above features have all expressed considerable comfort with them and are ready to stand-by to help expedite any necessary bug-fixes etc. to get to stabilization quickly. I'm confident we can get this release out by end of May. This sets stage for a hadoop-2.x GA release right after with some more testing - this means I think I can quickly turn around and make bug-fix releases as necessary right after 2.0.5-beta. I request that people consider helping out with this plan and sign up to help push hadoop-2.x to stability as outlined above. I believe this will help achieve our shared goals of quickly stabilizing hadoop-2 and help ensure we can support it for forseeable future in a compatible manner for the benefit of our users and downstream projects. Please vote, the vote will run the normal 7 days. Obviously, I'm +1. thanks, Arun PS: To keep this discussion grounded in technical details I've moved this to dev@ (bcc general@). -- http://hortonworks.com/download/
Re: [VOTE] - Release 2.0.5-beta
On Wed, May 15, 2013 at 9:25 PM, Andrew Purtell apurt...@apache.org wrote: The other thread or vote or whatever at least served the purpose in fresh surfacing of concerns. Talk of new features going in to a beta on a very short short timetable is concerning for anyone with experience working on large software projects. It's not a little ironic that this vote thread, done in response to sort out the other one predicated on stability concerns, begins with a laundry list of features and JIRAs to go in. I think it is usually the case that a beta release receives only bugfixes* over the alpha that proceeded it. This may just be a lack of consensus on what beta means. Assuming that you are talking about HDFS features when you say features going into a beta on a very short short timetable and laundry list etc, I request you to take a cursory look at the development of these features. Snapshot is being developed since 2012 Nov, excluding the early prototype that happened in 2012 May. Most of the development was complete by the early February except for the support of rename capability, which has been tricky. As regards to Windows support, this is a work that has been happening for more than an year in many other branches. So these features are not something that are impulsively developed and irresponsibly pushed to a release. They have gone through considerable testing and have been developed over a long time. Please set aside discussion on particular features or Hadoop bylaws or politics or debate club. I can't speak for all of downstream of course, but to the extent that I can I can say we don't care about that. The core ask, at least mine, is take a fresh look at reducing per-release disruptions to the rest of the entire ecosystem that has grown up around Hadoop. What is the disruption you anticipate due to the current content of the release? If it is stability, I am confident that very few bugs will come out of these features and stability should not be affected. This has been the case for the HDFS features for many years. The development is generally done in a feature branch, the feature is tested and stabilized in that branch before merging to trunk. This is contrary to few people's incorrect claims about how it has taken a long time to stabilize an HDFS features in branch-2. Needless to say stability is not just a concern of downstream projects. We spend long hours, day in day out, trying to ensure features are stable as core contributors. Regards, Suresh
Re: [VOTE] Hadoop release candidate 1.2.0-rc1
Matt, Thanks for getting the release out. I downloaded the tarballs and verified the checksums. I ran the following tests on a single node cluster: - Ensured a newly installed cluster can be started and is functional. Also checked the cluster restarts. - Checked webUI for HDFS and MapReduce. - Tested some common file system CLIs on HDFS. - Ran TestDFSIO, NNBench jobs. +1 (binding) for the release. Regards, Suresh On Fri, May 10, 2013 at 9:44 AM, Matt Foley mfo...@hortonworks.com wrote: Hi all, just a reminder this vote is underway and will close Monday 11:30am. Please review and vote! Thanks, --Matt On Mon, May 6, 2013 at 11:36 AM, Matt Foley ma...@apache.org wrote: Friends, Nexus issues are resolved, and the Nexus staging repository for Hadoop 1.2.0-rc1 properly uploaded. Thanks for your patience. --Matt On Mon, May 6, 2013 at 11:11 AM, Matt Foley ma...@apache.org wrote: Hi all, I have posted the signed tarballs for Hadoop 1.2.0-rc1 at http://people.apache.org/~mattf/hadoop-1.2.0-rc1/ Release notes are at: releasenotes_1.2.0-rc1.html http://people.apache.org/~mattf/hadoop-1.2.0-rc1/releasenotes_1.2.0-rc1.html I'm having a little trouble with Nexus (it seems to have forgotten I exist) but am working on that and will post to Nexus as soon as possible. In the meantime, unless there are objections, I'd like to start the vote. Please review this release candidate and vote it for release. Vote will end in seven days as usual, at 11:30am PDT on Monday 13 May. Best regards, --Matt (release manager) -- http://hortonworks.com/download/
[jira] [Resolved] (HADOOP-9537) Backport AIX patches to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-9537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9537. - Resolution: Fixed Fix Version/s: (was: 1.3.0) 1.2.0 Hadoop Flags: Reviewed I committed the patch to branch-1 and branch-1.2. Backport AIX patches to branch-1 Key: HADOOP-9537 URL: https://issues.apache.org/jira/browse/HADOOP-9537 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 1.3.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: HADOOP-9537.001.patch Backport couple of trivial Jiras to branch-1. HADOOP-9305 Add support for running the Hadoop client on 64-bit AIX HADOOP-9283 Add support for running the Hadoop client on AIX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9472) Cleanup hadoop-config.cmd
[ https://issues.apache.org/jira/browse/HADOOP-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9472. - Resolution: Fixed I committed this patch to branch-1-win. Thank you Ivan. Cleanup hadoop-config.cmd - Key: HADOOP-9472 URL: https://issues.apache.org/jira/browse/HADOOP-9472 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Priority: Minor Attachments: HADOOP-9472.branch-1-win.cleanup.2.patch, HADOOP-9472.branch-1-win.cleanup.patch Some portions of hadoop-config.cmd script are unused and should be cleaned up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Heads up - 2.0.5-beta
Eli, I will post a more detailed reply soon. But one small correction: I'm also not sure there's currently consensus on what an incompatible change is. For example, I think HADOOP-9151 is incompatible because it broke client/server wire compatibility with previous releases and any change that breaks wire compatibility is incompatible. Suresh felt it was not an incompatible change because it did not affect API compatibility (ie PB is not considered part of the API) and the change occurred while v2 is in alpha. This is not correct. I did not say it was not an incompatible change. It was indeed an incompatible wire protocol change. My argument was, the phase of development we were in, we could not mark wire protocol as stable and not make any incompatible change. But once 2.0.5-beta is out, as had discussed earlier, we should not make further incompatible changes to wire protocol. -- http://hortonworks.com/download/
Re: Heads up - 2.0.5-beta
Thanks for starting this discussion. I volunteer to do a final review of protocol changes, so we can avoid incompatible changes to API and wire protocol post 2.0.5 in Common and HDFS. We have been working really hard on the following features. I would like to get into 2.x and see it reach HDFS users: # Snapshots # NFS gateway for HDFS # HDFS-347 unix domain socket based short circuits # Windows support Other HDFS folks please let me know if I missed anything. To ensure a timely release of 2.0.5-beta, we should not hold back for individual features. However, I would like to make necessary API and/or protocol changes right-away. This will allow us to adding features in subsequent releases e.g. hadoop-2.2 or hadoop-2.3 etc without breaking compatibility. For e.g. even if we don't complete NFS support, making FileID related changes in 2.0.5-beta will ensure future compatbility. Regards, Suresh On Thu, Apr 25, 2013 at 6:34 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, With hadoop-2.0.4-alpha released, I'd like 2.0.4 to be the final of our hadoop-2.x alphas. We have made lots of progress on hadoop-2.x and I believe we are nearly there, exciting times! As we have discussed previously, I hope to do a final push to stabilize hadoop-2.x, release a hadoop-2.0.5-beta in the next month or so; and then declare hadoop-2.1 as stable this summer after a short period of intensive testing. With that in mind, I really want to make a serious push to lock down APIs and wire-protocols for hadoop-2.0.5-beta. Thus, we can confidently support hadoop-2.x in a compatible manner in the future. So, it's fine to add new features, but please ensure that all APIs are frozen for hadoop-2.0.5-beta Vinod is helping out on the YARN/MR side and has tagged a number of final changes (including some the final API incompatibilities) we'd like to push in before we call hadoop-2.x as ready to be supported (Target Version set to 2.0.5-beta): http://s.apache.org/target-hadoop-2.0.5-beta Thanks Vinod! (Note some of the sub-tasks of umbrella jiras may not be tagged, but their necessity is implied). Similarly on HDFS side, can someone please help out by tagging features, bug-fixes, protocol/API changes etc.? This way we can ensure HDFS APIs protocols are locked down too - I'd really appreciate it! thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- http://hortonworks.com/download/
[jira] [Resolved] (HADOOP-9480) The windows installer should pick the config from src\packages\win\template\conf
[ https://issues.apache.org/jira/browse/HADOOP-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9480. - Resolution: Fixed Hadoop Flags: Reviewed I committed the patch to branch-1-win. Thank you Ivan. Thanks to Mostafa for the review. The windows installer should pick the config from src\packages\win\template\conf Key: HADOOP-9480 URL: https://issues.apache.org/jira/browse/HADOOP-9480 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9480.branch-1-win.patch We should pick the config files from the src\packages\win\template\conf location rather than the conf\ location in the Windows installer. Reported by [~mostafae]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9452) Windows install scripts bugfixes
[ https://issues.apache.org/jira/browse/HADOOP-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9452. - Resolution: Fixed Hadoop Flags: Reviewed I committed the patch to trunk. Thank you Ivan and Kanna. Windows install scripts bugfixes Key: HADOOP-9452 URL: https://issues.apache.org/jira/browse/HADOOP-9452 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9452.branch-1-win.installerfixes.2.patch, HADOOP-9452.branch-1-win.installerfixes.patch A few bugfixes we've done to install scripts on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9071) configure ivy log levels for resolve/retrieve
[ https://issues.apache.org/jira/browse/HADOOP-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9071. - Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed Committed the patch to branch-1 and 1.2. Thank you Giri. Thank you Chris for the review and testing. configure ivy log levels for resolve/retrieve - Key: HADOOP-9071 URL: https://issues.apache.org/jira/browse/HADOOP-9071 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 1.1.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 1.2.0 Attachments: HADOOP-9071.PATCH, HADOOP-9071-V1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9253) Capture ulimit info in the logs at service start time
[ https://issues.apache.org/jira/browse/HADOOP-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9253. - Resolution: Fixed Target Version/s: 2.0.3-alpha, 1.2.0 (was: 1.2.0, 2.0.3-alpha) Capture ulimit info in the logs at service start time - Key: HADOOP-9253 URL: https://issues.apache.org/jira/browse/HADOOP-9253 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Arpit Gupta Assignee: Arpit Gupta Fix For: 1.2.0, 0.23.7, 2.0.5-beta Attachments: HADOOP-9253.branch-1.patch, HADOOP-9253.branch-1.patch, HADOOP-9253.branch-1.patch, HADOOP-9253.patch, HADOOP-9253.patch output of ulimit -a is helpful while debugging issues on the system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9442) Splitting issue when using NLineInputFormat with compression
[ https://issues.apache.org/jira/browse/HADOOP-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9442. - Resolution: Invalid Jira is for reporting bugs and not for asking bugs. Please use the user mailing lists for questions such as this. Splitting issue when using NLineInputFormat with compression Key: HADOOP-9442 URL: https://issues.apache.org/jira/browse/HADOOP-9442 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.1.2 Environment: Try in Apache Hadoop 1.1.1, CDH4, and Amazon EMR. Same result. Reporter: Qiming He Priority: Minor #make a long text line. It seems only long line text causing issue. $ cat abook.txt | base64 –w 0 onelinetext.b64 #200KB+ long $ hadoop fs –put onelinetext.b64 /input/onelinetext.b64 $ hadoop jar hadoop-streaming.jar \ -input /input/onelinetext.b64 \ -output /output \ -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat \ –mapper wc Num task: 1, and output has one line: Line 1: 1 2 202699 which makes sense because one line per mapper is intended. Then, using compression with NLineInputFormat $ bzip2 onelinetext.b64 $ hadoop fs –put onelinetext.b64.bz2 /input/onelinetext.b64.bz2 $ hadoop jar hadoop-streaming.jar \ -Dmapred.input.compress=true \ -Dmapred.input.compression.codec=org.apache.hadoop.io.compress.GzipCodec \ -input /input/onelinetext.b64.gz \ -output /output \ -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat \ –mapper wc I am expecting the same results as above, 'coz decompressing should occur before processing one-line text (i.e. wc), however, I am getting: Num task: 397 (or other large numbers depend on environments), and output has 397 lines: Line1-396: 0 0 0 Line 397: 1 2 202699 Any idea why so many mapred.map.tasks 1? Is it incorrect splitting? I purposely choose gzip because I believe it is NOT split-able. I got similar results when using bzip2 and lzop codecs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Adding other mailing lists I missed earlier. Cos, There is progress being made on that ticket. Also it has nothing to do with that. Please follow the discussion here and why this happened due to an invalid commit that was reverted - https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650 Regards, Suresh On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik c...@apache.org wrote: It doesn't look like any progress has been done on the ticket below in the last 3 weeks. And now branch-2 can't be compiled because of hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from outside package That's exactly why I was -1'ing this... Cos On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: Thanks, gentlemen. I've opened and taken responsibility for https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has agreed to help with the parts that require Jenkins admin access. Thanks, --Matt On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko shv.had...@gmail.comwrote: +1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote:
[jira] [Resolved] (HADOOP-9408) misleading description for net.topology.table.file.name property in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9408.
-
Resolution: Fixed
Fix Version/s: 2.0.5-beta
Committed to both trunk and branch-2. Thank you Rajesh.
misleading description for net.topology.table.file.name property in
core-default.xml
Key: HADOOP-9408
URL: https://issues.apache.org/jira/browse/HADOOP-9408
Project: Hadoop Common
Issue Type: Bug
Components: conf
Reporter: rajeshbabu
Priority: Minor
Fix For: 2.0.5-beta
Attachments: HAHOOP_9408.patch
property
namenet.topology.table.file.name/name
value/value
description The file name for a topology file, which is used when the
*net.topology.script.file.name* property is set to
org.apache.hadoop.net.TableMapping. The file format is a two column text
file, with columns separated by whitespace. The first column is a DNS or
IP address and the second column specifies the rack where the address
maps.
If no entry corresponding to a host in the cluster is found, then
/default-rack is assumed.
/description
/property
Marked property name(net.topology.script.file.name) should be
{code}
net.topology.node.switch.mapping.impl
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [PROPOSAL] Hadoop branch-1.2
Steve, is there a timeline all these changes will be ready? If it is not going to be ready soon, perhaps these changes could be consider for 1.3? On Sat, Mar 9, 2013 at 2:38 PM, Matt Foley mfo...@hortonworks.com wrote: Steve, I'm okay with this, but HADOOP-9258https://issues.apache.org/jira/browse/HADOOP-9258 is dependent on HADOOP-9265 https://issues.apache.org/jira/browse/HADOOP-9265, which itself seems ready to commit. As I read the comments, both are ready to commit. Please let me know if this is correct. In meantime, I'm going to make the branch. We'll just have to commit to both branch-1 and branch-1.2. Thanks, --Matt On Thu, Mar 7, 2013 at 1:29 AM, Steve Loughran ste...@hortonworks.com wrote: On 6 March 2013 23:17, Matt Foley ma...@apache.org wrote: Hi, I got stuck in other work and did not make the Hadoop 1.2 branch in February. Now that release 1.1.2 is out, I'm ready to make the 1.2 branch. I intend to branch for 1.2 in the next night or two, and at that point will make the formal proposal to schedule an rc a week after that. If any issues, please let me know. Regards, --Matt I'd like to get my FS contract tests in there with the fixes for s3n and s3 to avoid data loss if you try to move a directory under itself. https://issues.apache.org/jira/browse/HADOOP-9258 -- http://hortonworks.com/download/
[jira] [Resolved] (HADOOP-8796) commands_manual.html link is broken
[ https://issues.apache.org/jira/browse/HADOOP-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8796. - Resolution: Not A Problem Resolving the bug as Not A Problem. commands_manual.html link is broken --- Key: HADOOP-8796 URL: https://issues.apache.org/jira/browse/HADOOP-8796 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.0.1-alpha Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Priority: Minor Attachments: screenshot-1.jpg If you go to http://hadoop.apache.org/docs/r2.0.0-alpha/ and click on Hadoop Commands you are getting a broken link: http://hadoop.apache.org/docs/r2.0.0-alpha/hadoop-project-dist/hadoop-common/commands_manual.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9326) maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-common: There a test failures.
[ https://issues.apache.org/jira/browse/HADOOP-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9326. - Resolution: Invalid maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-common: There a test failures. - Key: HADOOP-9326 URL: https://issues.apache.org/jira/browse/HADOOP-9326 Project: Hadoop Common Issue Type: Bug Components: build, test Environment: For information, i take hadoop with GIT and i run it on mac OS Reporter: JLASSI Aymen Original Estimate: 336h Remaining Estimate: 336h I'd like to compile hadoop from source code, and when i launch test-step, i have the desciption as follows, when i skip the test-step to the package step, i have the same problem, the same description of bug: Results : Failed tests: testFailFullyDelete(org.apache.hadoop.fs.TestFileUtil): The directory xSubDir *should* not have been deleted. expected:true but was:false testFailFullyDeleteContents(org.apache.hadoop.fs.TestFileUtil): The directory xSubDir *should* not have been deleted. expected:true but was:false testListStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem): Should throw IOException test0[0](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for build/test/temp/RELATIVE1 in build/test/temp/RELATIVE0/block4197707426846287299.tmp - FAILED! testROBufferDirAndRWBufferDir[0](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for build/test/temp/RELATIVE2 in build/test/temp/RELATIVE1/block138767728739012230.tmp - FAILED! testRWBufferDirBecomesRO[0](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for build/test/temp/RELATIVE3 in build/test/temp/RELATIVE4/block4888615109050601773.tmp - FAILED! test0[1](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block4663369813226761504.tmp - FAILED! testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block2846944239985650460.tmp - FAILED! testRWBufferDirBecomesRO[1](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE3 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE4/block4367331619344952181.tmp - FAILED! test0[2](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5687619346377173125.tmp - FAILED! testROBufferDirAndRWBufferDir[2](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block2235209534902942511.tmp - FAILED! testRWBufferDirBecomesRO[2](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED3 in /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED4/block6994640486900109274.tmp - FAILED! testReportChecksumFailure(org.apache.hadoop.fs.TestLocalFileSystem) testListStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem): Should throw IOException testCount(org.apache.hadoop.metrics2.util.TestSampleQuantiles): expected:50[.00 %ile +/- 5.00%: 1337(..) testCheckDir_notDir_local(org.apache.hadoop.util.TestDiskChecker): checkDir success testCheckDir_notReadable_local(org.apache.hadoop.util.TestDiskChecker): checkDir success testCheckDir_notWritable_local(org.apache.hadoop.util.TestDiskChecker
[jira] [Resolved] (HADOOP-9378) start_thrift_server can not run successfully.
[ https://issues.apache.org/jira/browse/HADOOP-9378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9378. - Resolution: Not A Problem start_thrift_server can not run successfully. - Key: HADOOP-9378 URL: https://issues.apache.org/jira/browse/HADOOP-9378 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 1.0.4 Reporter: KimboQi Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration, then i add for f in $TOP/*.jar ; do CLASSPATH=$CLASSPATH:$f done into the script, it run successfully -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9373) Merge CHANGES.branch-trunk-win.txt to CHANGES.txt
Suresh Srinivas created HADOOP-9373: --- Summary: Merge CHANGES.branch-trunk-win.txt to CHANGES.txt Key: HADOOP-9373 URL: https://issues.apache.org/jira/browse/HADOOP-9373 Project: Hadoop Common Issue Type: Bug Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor This is to merge the changes from CHANGES.branch-trunk-win.txt to appropriate CHANGES.txt files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Thank you all for voting and participating in the discussions. With 11 +1s from committers (more than the required 3 +1s from active committers per the Hadoop bylaws), 1 +0, 8 +1s from other contributors, and no -1s the merge vote passes. I have committed the consolidated patch from branch-trunk-win to trunk. People who are interested in following up on the progress related to Jenkins setup for Windows and other work that came up during the discussion, please watch: https://issues.apache.org/jira/browse/HADOOP-9359 Regards, Suresh On Mon, Mar 4, 2013 at 5:41 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks, gentlemen. I've opened and taken responsibility for https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has agreed to help with the parts that require Jenkins admin access. Thanks, --Matt On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko shv.had...@gmail.com wrote: +1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration (pre-commit build and unit test triggered by Jira Patch Available state) satisfy your request for functionality #1 and #2? Yes or no, please. Thanks, --Matt On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantin, I would like to explore what it would take to remove this perceived impediment -- Glad you decided to explore. Thank you. although I reserve the right to argue that this is not pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede
[jira] [Resolved] (HADOOP-9368) Add timeouts to new tests in branch-trunk-win
[ https://issues.apache.org/jira/browse/HADOOP-9368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9368. - Resolution: Fixed Hadoop Flags: Reviewed +1. I committed the patch. Thank you Arpit for the quick turnaround. I will post a merge patch again to see how the consolidated patch fares with Jenkins. Add timeouts to new tests in branch-trunk-win - Key: HADOOP-9368 URL: https://issues.apache.org/jira/browse/HADOOP-9368 Project: Hadoop Common Issue Type: Bug Affects Versions: trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: trunk-win Attachments: HADOOP-9368.patch, HADOOP-9368.patch, HADOOP-9368.patch Add timeouts to the new tests so they can be integrated into trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
On Sun, Mar 3, 2013 at 8:50 PM, Harsh J ha...@cloudera.com wrote: Have we agreed (and stated it somewhere proper) that a -1 obtained for a Windows CI build for a test-patch will not block the ongoing work (unless it is Windows specific) and patches may still be committed to trunk despite that? This thread is long and possibly hard to follow. Yes, I and several others have stated that for now it is okay to commit even if Windows precommit build posts -1. I'm +1 if someone can assert and add the above into the formal guidelines. I'd still prefer that Windows does its releases separately as that ensures more quality for its audience and better testing periods (and wouldn't block anything), but we can come to that iff we are unable to maintain the currently proposed model. Which do you think is the right place to add this? At this time we are voting for merging into trunk. I prefer having a single release that supports both Linux and windows. Based on working on Windows support I think this is doable and should not hold up releases for Linux.
[jira] [Resolved] (HADOOP-9354) Windows native project files missing license headers
[ https://issues.apache.org/jira/browse/HADOOP-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9354. - Resolution: Fixed Hadoop Flags: Reviewed +1. I committed the patch to branch-trunk-win. Thank you Chris! Windows native project files missing license headers Key: HADOOP-9354 URL: https://issues.apache.org/jira/browse/HADOOP-9354 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial Attachments: HADOOP-9354-branch-trunk-win.1.patch, HADOOP-9354-branch-trunk-win.2.patch We need to add the license header to native.sln, native.vcxproj, and native.vcxproj.filters. The equivalent files in winutils already have the license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9356) remove remaining references to cygwin/cygpath from scripts
[ https://issues.apache.org/jira/browse/HADOOP-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9356. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch to branch-trunk-win. Thank you Chris! remove remaining references to cygwin/cygpath from scripts -- Key: HADOOP-9356 URL: https://issues.apache.org/jira/browse/HADOOP-9356 Project: Hadoop Common Issue Type: Improvement Components: build, scripts Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9356-branch-trunk-win.1.patch, HADOOP-9356-branch-trunk-win.2.patch branch-trunk-win still contains a few references to Cygwin and the cygpath command that need to be removed now that they are no longer needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9232) JniBasedUnixGroupsMappingWithFallback fails on Windows with UnsatisfiedLinkError
[
https://issues.apache.org/jira/browse/HADOOP-9232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9232.
-
Resolution: Fixed
Fix Version/s: trunk-win
Hadoop Flags: Reviewed
Committed the patch to the branch.
Thank you Ivan!. Thank you Arpit, Chuan and Chris for the review.
JniBasedUnixGroupsMappingWithFallback fails on Windows with
UnsatisfiedLinkError
Key: HADOOP-9232
URL: https://issues.apache.org/jira/browse/HADOOP-9232
Project: Hadoop Common
Issue Type: Bug
Components: native, security
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Ivan Mitic
Fix For: trunk-win
Attachments: HADOOP-9232.branch-trunk-win.jnigroups.2.patch,
HADOOP-9232.branch-trunk-win.jnigroups.3.patch,
HADOOP-9232.branch-trunk-win.jnigroups.patch, HADOOP-9232.patch
{{JniBasedUnixGroupsMapping}} calls native code which isn't implemented
properly for Windows, causing {{UnsatisfiedLinkError}}. The fallback logic
in {{JniBasedUnixGroupsMappingWithFallback}} works by checking if the native
code is loaded during startup. In this case, hadoop.dll is present and
loaded, but it doesn't contain the right code. There will be no attempt to
fallback to {{ShellBasedUnixGroupsMapping}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9250) Windows installer bugfixes
[ https://issues.apache.org/jira/browse/HADOOP-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9250. - Resolution: Fixed Fix Version/s: 1-win Hadoop Flags: Reviewed +1. I committed the patch to branch-1-win. Thank you Ivan! Thanks to [~kanna...@microsoft.com] for the review. Windows installer bugfixes -- Key: HADOOP-9250 URL: https://issues.apache.org/jira/browse/HADOOP-9250 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 1-win Attachments: HADOOP-9250.branch-1-win.installerbugs.patch A few bugfixes and improvements we made to the install scripts on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9347) add instructions to BUILDING.txt describing how to build on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9347. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch to branch-trunk-win. Thank you Chris! Thank you Arpit for the review. add instructions to BUILDING.txt describing how to build on Windows --- Key: HADOOP-9347 URL: https://issues.apache.org/jira/browse/HADOOP-9347 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9347-branch-trunk-win.1.patch, HADOOP-9347-branch-trunk-win.2.patch, HADOOP-9347-branch-trunk-win.3.patch Add documentation to BUILDING.txt describing dependencies and instructions for building on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Thanks for raising good questions. Currently the merge patch passes all the tests on Linux, hence the proposal for merging the patch to trunk. But as Bobby, Harsh and Eli pointed out, before declaring support for Windows, we need the discussion on the following: 1. Precommit and development process Jenkins infrastructure for Windows build will be made available. Giri and Microsoft contributors have volunteered to help make this happen. With that we need to decide how our precommit process looks. My inclination is to wait for +1 from precommit builds on both the platforms to ensure no issues are introduced. Thoughts? 2. Feature development impact Some questions have been raised about would new features need to be supported on both the platforms. Yes. I do not see a reason why features cannot work on both the platforms, with the exception of platform specific optimizations. This what Java gives us. 3. Platform specific features/optimizations As regards platform specific optimization, each platform can evolve at its own pace and should not block progress of a specific platform. As indicated in my earlier email, there is a sizable number of contributors to work on issues and support of Hadoop on Windows platform. I am excited to see Hadoop reach the other large part of server market. Eli, as pointed out by you, the TODO items need to be addressed. Also we realized we still need to add information on how to build on Windows in BUILDING.txt. We will address this ASAP. Giri and Matt have some expirience with this and should be able to provide more information. On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins e...@cloudera.com wrote: Bobby raises some good questions. A related one, since most current developers won't add Windows support for new features that are platform specific is it assumed that Windows development will either lag or will people actively work on keeping Windows up with the latest? And vice versa in case Windows support is implemented first. Is there a jira for resolving the outstanding TODOs in the code base (similar to HDFS-2148)? Looks like this merge doesn't introduce many which is great (just did a quick diff and grep). Thanks, Eli On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans ev...@yahoo-inc.com wrote: After this is merged in is Windows still going to be a second class citizen but happens to work for more than just development or is it a fully supported platform where if something breaks it can block a release? How do we as a community intend to keep Windows support from breaking? We don't have any Jenkins slaves to be able to run nightly tests to validate everything still compiles/runs. This is not a blocker for me because we often rely on individuals and groups to test Hadoop, but I do think we need to have this discussion before we put it in. --Bobby On 2/26/13 4:55 PM, Suresh Srinivas sur...@hortonworks.com wrote: I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I am happy to announce that we are ready for the merge. Here is a brief recap on the highlights of the work done: - Command-line scripts for the Hadoop surface area - Mapping the HDFS permissions model to Windows - Abstracted and reconciled mismatches around differences in Path semantics in Java and Windows - Native Task Controller for Windows - Implementation of a Block Placement Policy to support cloud environments, more specifically Azure. - Implementation of Hadoop native libraries for Windows (compression codecs, native I/O) - Several reliability issues, including race-conditions, intermittent test failures, resource leaks. - Several new unit test cases written for the above changes Please find the details of the work in CHANGES.branch-trunk-win.txt - Common changeshttp://bit.ly/Xe7Ynv, HDFS changeshttp://bit.ly/13QOSo9 , and YARN and MapReduce changes http://bit.ly/128zzMt. This is the work ported from branch-1-win to a branch based on trunk. For details of the testing done, please see the thread - http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562 https://issues.apache.org/jira/browse/HADOOP-8562. This was a large undertaking that involved developing code, testing the entire Hadoop stack, including scale tests. This is made possible only with the contribution from many many folks in the community. Following people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who contributed as well providing feedback and comments on numerous jiras. The vote will run for seven days and will end on March 5, 6:00PM PST
[jira] [Resolved] (HADOOP-9279) Document the need to build hadoop-maven-plugins for eclipse and separate project builds
[
https://issues.apache.org/jira/browse/HADOOP-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9279.
-
Resolution: Fixed
Fix Version/s: 2.0.4-beta
Hadoop Flags: Reviewed
I committed the patch to trunk and branch-2.
Thank you Tsuyoshi! Thank you Chris for the review.
Document the need to build hadoop-maven-plugins for eclipse and separate
project builds
---
Key: HADOOP-9279
URL: https://issues.apache.org/jira/browse/HADOOP-9279
Project: Hadoop Common
Issue Type: Improvement
Components: build, documentation
Affects Versions: 3.0.0
Environment: hadoop-maven-plugins-3.0.0-SNAPSHOT.jar doesn't exist in
local repository or at maven central repository.
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Fix For: 2.0.4-beta
Attachments: HADOOP-9279.2.patch, HADOOP-9279.3.patch,
HADOOP-9279.patch
In current hadoop-trunk, compile fails when
hadoop-maven-plugins-3.0.0-SNAPSHOT.jar doesn't exist in local repository or
at maven central repository.
The affected components are:
./hadoop-common-project/hadoop-common/pom.xml
./hadoop-maven-plugins/pom.xml
./hadoop-project/pom.xml
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
The error log is as follows
{quote}
[ERROR] Plugin org.apache.hadoop.maven.plugin:hadoop-maven-plugins:1.0 or one
of its dependencies could not be resolved: Failed to read artifact descriptor
for org.apache.hadoop.maven.plugin:hadoop-maven-plugins:jar:1.0: Could not
find artifact org.apache.hadoop.maven.plugin:hadoop-maven-plugins:pom:1.0 in
central (http://repo.maven.apache.org/maven2) - [Help 1]
{quote}
This can be avoidable if hadoop-maven-plugins is installed before packaging.
This is undocumented, so this should get documented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9313) Remove spurious mkdir from hadoop-config.cmd
[
https://issues.apache.org/jira/browse/HADOOP-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9313.
-
Resolution: Fixed
Fix Version/s: trunk-win
Hadoop Flags: Reviewed
+1 for the patch. I committed it to branch-trunk-win.
Thank you Ivan!
Remove spurious mkdir from hadoop-config.cmd
Key: HADOOP-9313
URL: https://issues.apache.org/jira/browse/HADOOP-9313
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: trunk-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: trunk-win
Attachments: HADOOP-9313.branch-trunk-win.cmd.patch
The following mkdir seems to have been accidentally added to Windows cmd
script and should be removed:
{code}
mkdir c:\tmp\dir1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9309) test failures on Windows due to UnsatisfiedLinkError in NativeCodeLoader#buildSupportsSnappy
[
https://issues.apache.org/jira/browse/HADOOP-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9309.
-
Resolution: Fixed
Fix Version/s: trunk-win
Hadoop Flags: Reviewed
I committed the patch to branch-trunk-win. Thank you Aprit!
Thank you Chris for the review.
test failures on Windows due to UnsatisfiedLinkError in
NativeCodeLoader#buildSupportsSnappy
Key: HADOOP-9309
URL: https://issues.apache.org/jira/browse/HADOOP-9309
Project: Hadoop Common
Issue Type: Bug
Components: native, util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Arpit Agarwal
Fix For: trunk-win
Attachments: HADOOP-9309.1.patch, HADOOP-9309.patch
Checking for Snappy support calls native method
{{NativeCodeLoader#buildSupportsSnappy}}. This method has not been
implemented for Windows in hadoop.dll, so it throws {{UnsatisfiedLinkError}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Heads up - merge branch-trunk-win to trunk
I was planning to add details about the testing in the subsequent voting thread. As pointed out in the emails above, a lot of work is related to handling difference between the platforms related to paths and utilities. These changes lend themselves to be tested very well by the existing unit tests. That said, a lot of testing has happened to validate the development done in these branches. On branch-1-win, Hortonworks QA has been running a suite of comprehensive system tests. This involves testing the branch with all the other stack components, such as Pig, Hive, HCat, HBase and Oozie. These tests have been done both on Linux and Windows. The testing has also been done both on JDK 6 and 7. In addition, as Mahadevan pointed out it has also been tested by early customers with production loads and at scale. branch-trunk-win has lot of code in common with branch-1-win and hence benefits from all the above testing. One place where additional work and testing was done in branch-trunk-win is related to YARN. All the MapReduce related work loads have been validated on single node cluster, cluster sizes with nodes upwards of 10. On Thu, Feb 7, 2013 at 6:46 PM, Eli Collins e...@cloudera.com wrote: Thanks for the update Suresh. Has any testing been done on the branch on Linux aside from running the unit tests? Thanks, Eli On Thu, Feb 7, 2013 at 5:42 PM, Suresh Srinivas sur...@hortonworks.com wrote: The support for Hadoop on Windows was proposed in HADOOP-8079https://issues.apache.org/jira/browse/HADOOP-8079 almost a year ago. The goal was to make Hadoop natively integrated, full-featured, and performance and scalability tuned on Windows Server or Windows Azure. We are happy to announce that a lot of progress has been made in this regard. Initial work started in a feature branch, branch-1-win, based on branch-1. The details related to the work done in the branch can be seen in CHANGES.txt http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup . This work has been ported to a branch, branch-trunk-win, based on trunk. Merge patch for this is available on HADOOP-8562https://issues.apache.org/jira/browse/HADOOP-8562 . Highlights of the work done so far: 1. Necessary changes in Hadoop to run natively on Windows. These changes handle differences in platforms related to path names, process/task management etc. 2. Addition of winutils tools for managing file permissions and ownership, user group mapping, hardlinks, symbolic links, chmod, disk utilization, and process/task management. 3. Added cmd scripts equivalent to existing shell scripts hadoop-daemon.sh, start and stop scripts. 4. Addition of block placement policy implemnation to support cloud enviroment, more specifically Azure. We are very close to wrapping up the work in branch-trunk-win and getting ready for a merge. Currently the merge patch is passing close to 100% of unit tests on Linux. Soon I will call for a vote to merge this branch into trunk. Next steps: 1. Call for vote to merge branch-trunk-win to trunk, when the work completes and precommit build is clean. 2. Start a discussion on adding Jenkins precommit builds on windows and how to integrate that with the existing commit process. Let me know if you have any questions. Regards, Suresh -- http://hortonworks.com/download/
Heads up - merge branch-trunk-win to trunk
The support for Hadoop on Windows was proposed in HADOOP-8079https://issues.apache.org/jira/browse/HADOOP-8079 almost a year ago. The goal was to make Hadoop natively integrated, full-featured, and performance and scalability tuned on Windows Server or Windows Azure. We are happy to announce that a lot of progress has been made in this regard. Initial work started in a feature branch, branch-1-win, based on branch-1. The details related to the work done in the branch can be seen in CHANGES.txthttp://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup. This work has been ported to a branch, branch-trunk-win, based on trunk. Merge patch for this is available on HADOOP-8562https://issues.apache.org/jira/browse/HADOOP-8562 . Highlights of the work done so far: 1. Necessary changes in Hadoop to run natively on Windows. These changes handle differences in platforms related to path names, process/task management etc. 2. Addition of winutils tools for managing file permissions and ownership, user group mapping, hardlinks, symbolic links, chmod, disk utilization, and process/task management. 3. Added cmd scripts equivalent to existing shell scripts hadoop-daemon.sh, start and stop scripts. 4. Addition of block placement policy implemnation to support cloud enviroment, more specifically Azure. We are very close to wrapping up the work in branch-trunk-win and getting ready for a merge. Currently the merge patch is passing close to 100% of unit tests on Linux. Soon I will call for a vote to merge this branch into trunk. Next steps: 1. Call for vote to merge branch-trunk-win to trunk, when the work completes and precommit build is clean. 2. Start a discussion on adding Jenkins precommit builds on windows and how to integrate that with the existing commit process. Let me know if you have any questions. Regards, Suresh
[jira] [Resolved] (HADOOP-9266) correct javac, findbugs, and release audit warnings on branch-trunk-win
[ https://issues.apache.org/jira/browse/HADOOP-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9266. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed Committed the patch to branch-trunk-win. Thank you Chris! correct javac, findbugs, and release audit warnings on branch-trunk-win --- Key: HADOOP-9266 URL: https://issues.apache.org/jira/browse/HADOOP-9266 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9266-branch-trunk-win.1.patch There are several new release audit warnings reported on branch-trunk-win for files introduced during the Windows compatibility work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9245) mvn clean without running mvn install before fails
[ https://issues.apache.org/jira/browse/HADOOP-9245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9245. - Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed I committed the patch to trunk and branch-trunk-win. Thank you Karthik! mvn clean without running mvn install before fails -- Key: HADOOP-9245 URL: https://issues.apache.org/jira/browse/HADOOP-9245 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0, trunk-win Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 3.0.0 Attachments: HADOOP-9245.patch HADOOP-8924 introduces plugin dependency on hadoop-maven-plugins in hadoop-common and hadoop-yarn-common. Calling mvn clean on a fresh m2/repository (missing hadoop-maven-plugins) fails due to this dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8580) ant compile-native fails with automake version 1.11.3
[
https://issues.apache.org/jira/browse/HADOOP-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-8580.
-
Resolution: Fixed
Fix Version/s: 1.2.0
Hadoop Flags: Reviewed
I committed the patch to branch-1.
Thank you Gera for the patch.
ant compile-native fails with automake version 1.11.3
-
Key: HADOOP-8580
URL: https://issues.apache.org/jira/browse/HADOOP-8580
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.2.0, 0.22.0
Reporter: Eugene Koontz
Fix For: 1.2.0
Attachments: hadoop-8580-branch-1.gs.patch, HADOOP-8580-branch-1.patch
The following:
{code}
ant -d -v -DskipTests -Dcompile.native=true clean compile-native
{code}
works with GNU automake version 1.11.1, but fails with automake version
1.11.3.
Relevant lines of failure seem to be these:
{code}
[exec] make[1]: Leaving directory
`/tmp/hadoop-common/build/native/Linux-amd64-64'
[exec] Current OS is Linux
[exec] Executing 'sh' with arguments:
[exec] '/tmp/hadoop-common/build/native/Linux-amd64-64/libtool'
[exec] '--mode=install'
[exec] 'cp'
[exec] '/tmp/hadoop-common/build/native/Linux-amd64-64/libhadoop.la'
[exec] '/tmp/hadoop-common/build/native/Linux-amd64-64/lib'
[exec]
[exec] The ' characters around the executable and arguments are
[exec] not part of the command.
[exec] /tmp/hadoop-common/build/native/Linux-amd64-64/libtool: 3212:
/tmp/hadoop-common/build/native/Linux-amd64-64/libtool: install_prog+=cp: not
found
[exec] /tmp/hadoop-common/build/native/Linux-amd64-64/libtool: 3232:
/tmp/hadoop-common/build/native/Linux-amd64-64/libtool: files+=
/tmp/hadoop-common/build/native/Linux-amd64-64/libhadoop.la: not found
[exec] libtool: install: you must specify an install program
[exec] libtool: install: Try `libtool --help --mode=install' for more
information.
[antcall] Exiting /tmp/hadoop-common/build.xml.
BUILD FAILED
{code}
Created transcript showing entire output of the above ant command for both
automake 1.11.1 (successful) versus 1.11.3 (failed) here:
https://gist.github.com/3078988
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9197) Some little confusion in official documentation
[ https://issues.apache.org/jira/browse/HADOOP-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9197. - Resolution: Invalid Jason, for now I am going to resolve this jira as invalid, since you have not posted additional details. Feel free to open it when you have more concrete details/suggestions. Some little confusion in official documentation --- Key: HADOOP-9197 URL: https://issues.apache.org/jira/browse/HADOOP-9197 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Jason Lee Priority: Trivial Original Estimate: 336h Remaining Estimate: 336h I am just a newbie to Hadoop. recently i self-study hadoop. when i reading the official documentations, i find that them is a little confusion by beginners like me. for example, look at the documents about HDFS shell guide: In 0.17, the prefix of HDFS shell is hadoop dfs: http://hadoop.apache.org/docs/r0.17.2/hdfs_shell.html In 0.19, the prefix of HDFS shell is hadoop fs: http://hadoop.apache.org/docs/r0.19.1/hdfs_shell.html#lsr In 1.0.4,the prefix of HDFS shell is hdfs dfs: http://hadoop.apache.org/docs/r1.0.4/file_system_shell.html#ls As a beginner, i think reading them is suffering. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9191) TestAccessControlList and TestJobHistoryConfig fail with JDK7
[ https://issues.apache.org/jira/browse/HADOOP-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9191. - Resolution: Fixed Fix Version/s: (was: 1-win) Hadoop Flags: Reviewed I committed the patch to both branch-1 and branch-1-win. TestAccessControlList and TestJobHistoryConfig fail with JDK7 - Key: HADOOP-9191 URL: https://issues.apache.org/jira/browse/HADOOP-9191 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 1.2.0, 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: HADOOP-9191-branch-1.001.patch, HADOOP-9191-branch-1-win.001.patch, HADOOP-9191.patch Individual test cases have dependencies on a specific order of execution and fail when the order is changed. TestAccessControlList.testNetGroups relies on Groups being initialized with a hard-coded test class that subsequent test cases depend on. TestJobHistoryConfig.testJobHistoryLogging fails to shutdown the MiniDFSCluster on exit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9175) TestWritableName fails with Open JDK 7
[ https://issues.apache.org/jira/browse/HADOOP-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9175. - Resolution: Fixed Fix Version/s: 1.2.0 Target Version/s: (was: 1-win) Hadoop Flags: Reviewed +1 for the patch. Committed to branch-1. Thank you Arpit. TestWritableName fails with Open JDK 7 -- Key: HADOOP-9175 URL: https://issues.apache.org/jira/browse/HADOOP-9175 Project: Hadoop Common Issue Type: Test Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: HADOOP-9175.patch TestWritableName.testAddName fails due to a test order execution dependency on testSetName. java.io.IOException: WritableName can't load class: mystring at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73) at org.apache.hadoop.io.TestWritableName.testAddName(TestWritableName.java:92) Caused by: java.lang.ClassNotFoundException: mystring at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820) at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8106) hadoop-config.sh script defaults to /usr/etc/hadoop rather than /etc/hadoop for the default location of the conf dir
[ https://issues.apache.org/jira/browse/HADOOP-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8106. - Resolution: Duplicate HADOOP-8109 addresses this. hadoop-config.sh script defaults to /usr/etc/hadoop rather than /etc/hadoop for the default location of the conf dir Key: HADOOP-8106 URL: https://issues.apache.org/jira/browse/HADOOP-8106 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.0, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: HADOOP-8106.branch-1.0.patch, HADOOP-8106.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8981. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch. Thank you Xuan. TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Xuan Gong Fix For: trunk-win Attachments: HADOOP-8981-branch-trunk-win.1.patch, HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch, HADOOP-8981-branch-trunk-win.4.patch, HADOOP-8981-branch-trunk-win.5.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9081) Port TestWinUtils from branch-1-win to branch-trunk-win
[ https://issues.apache.org/jira/browse/HADOOP-9081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9081. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch to trunk-win. Thank you Chuan Liu, Ivan Mitic, Chris Nauroth, and Bikas Saha. Port TestWinUtils from branch-1-win to branch-trunk-win --- Key: HADOOP-9081 URL: https://issues.apache.org/jira/browse/HADOOP-9081 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9081-branch-trunk-win.1.patch, test-untar.tar, test-untar.tgz branch-1-win has a test suite named TestWinUtils to cover winutils.exe functionality. We need to port this to branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9056) Build native library on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9056. - Resolution: Fixed Hadoop Flags: Reviewed +1 for the patch. I committed the patch to branch-trunk-win. Thank you Chuan for the original branch-1-win patch and reviewing this patch. Thank you Arpit for porting all the related changes to branch-trunk-win. Build native library on Windows --- Key: HADOOP-9056 URL: https://issues.apache.org/jira/browse/HADOOP-9056 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: trunk-win Attachments: HADOOP-9056.1.patch, HADOOP-9056.3.patch, HADOOP-9056.5.patch, HADOOP-9056.6.patch, HADOOP-9056.7.patch, HADOOP-9056.patch Original Estimate: 168h Remaining Estimate: 168h The native library (hadoop.dll) must be compiled on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
[ https://issues.apache.org/jira/browse/HADOOP-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9102. - Resolution: Fixed Fix Version/s: 1-win Hadoop Flags: Reviewed +1. I committed the patch. Thank you Chris. winutils task isAlive does not return a non-zero exit code if the requested task is not alive - Key: HADOOP-9102 URL: https://issues.apache.org/jira/browse/HADOOP-9102 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 1-win Attachments: HADOOP-9102-branch-1-win.1.patch Work on YARN-233 noted that winutils task isAlive does not return a non-zero exit code if the requested task is not alive. This is inconsistent with the equivalent check on Unix, kill -0, which uses a non-zero exit code to communicate status. By changing winutils task isAlive to return a non-zero exit code, we can make the error handling code on the Java side consistent, avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9110) winutils ls off-by-one error indexing MONTHS array can cause access violation
[ https://issues.apache.org/jira/browse/HADOOP-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9110. - Resolution: Fixed Fix Version/s: trunk-win 1-win Hadoop Flags: Reviewed Patch committed to both branch-1-win and branch-trunk-win. Thank you Chris. winutils ls off-by-one error indexing MONTHS array can cause access violation - Key: HADOOP-9110 URL: https://issues.apache.org/jira/browse/HADOOP-9110 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 1-win, trunk-win Attachments: HADOOP-9110-branch-1-win.1.patch, HADOOP-9110-branch-trunk-win.1.patch In ls.c, the LsPrintLine function uses the wMonth field of a SYSTEMTIME struct to index into MONTHS, an array of 12 elements containing string representations of the months. The wMonth field is 1-based (1=January), but indexing into an array is zero-based. This gives the wrong month, and since we just crossed into month 12, it also has started causing an access violation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Hadoop Release 1.1.1
Andy, I committed MR-2374 to branch 1.1. It will be picked up if we release 1.1.2. On Wed, Nov 28, 2012 at 3:02 PM, Andy Isaacson a...@cloudera.com wrote: On Wed, Nov 28, 2012 at 2:52 PM, Matt Foley ma...@apache.org wrote: Andy, please commit MAPREDUCE-2374 to branch-1 and branch-1.1. That way it will be picked up by anyone who takes source code, and it will be in 1.1.2 and/or 1.2.0. I'm not a committer, could you merge the fix to branch-1.0 and branch-1.1? It's already on branch-1. It does not appear serious enough to invalidate the 1.1.1 release, altho I wish we had noticed it earlier. Fair enough. -andy -- http://hortonworks.com/download/
[jira] [Created] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
Suresh Srinivas created HADOOP-9094: --- Summary: Add interface audience and stability annotation to PathExceptions Key: HADOOP-9094 URL: https://issues.apache.org/jira/browse/HADOOP-9094 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks adding interface audience and stability to notation to those exceptions. It also tracks the comment from HADOOP-9093: bq. I propose using FileNotFoundException instead of PathNotFoundException as it is already extensively used. Similarly use AccessControlException instead of PathAccessException. If folks agree, I will make that change in the next patch. Alternatively we could at least make these exceptions subclasses of the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9058) branch-1-win build failure
[ https://issues.apache.org/jira/browse/HADOOP-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9058. - Resolution: Fixed branch-1-win build failure --- Key: HADOOP-9058 URL: https://issues.apache.org/jira/browse/HADOOP-9058 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1-win Reporter: Brandon Li Priority: Critical ... ... BUILD FAILED /Users/Brandon/git_trunk7/hadoop-common/build.xml:1284: The following error occurred while executing this line: /Users/Brandon/git_trunk7/hadoop-common/build.xml:1001: Warning: Could not find file /Users/Brandon/git_trunk7/hadoop-common/src/test/org/apache/hadoop/fs/test-untar.tgz to copy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9027) Build fails on Windows without sh/sed/echo in the path
[ https://issues.apache.org/jira/browse/HADOOP-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9027. - Resolution: Fixed Fix Version/s: 1-win Hadoop Flags: Reviewed +1. Committed the patch to branch-1-win. Thank you Ivan. Build fails on Windows without sh/sed/echo in the path -- Key: HADOOP-9027 URL: https://issues.apache.org/jira/browse/HADOOP-9027 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 1-win Attachments: HADOOP-9027.branch-1-win.cleanbuild.2.patch, HADOOP-9027.branch-1-win.cleanbuild.patch Branch-1-win still has a dependency on a few unix tools in compile time. Tracking Jira to remove this dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9011) saveVersion.py does not include branch in version annotation
[ https://issues.apache.org/jira/browse/HADOOP-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9011. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed +1. Committed to branch-trunk-win. Thank you Chris. Thank you Raja for the review. saveVersion.py does not include branch in version annotation Key: HADOOP-9011 URL: https://issues.apache.org/jira/browse/HADOOP-9011 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-9011-branch-trunk-win.patch HADOOP-8924 created saveVersion.py on branch-trunk-win. Unlike saveVersion.sh on trunk, it did not include the branch attribute in the version annotation. This causes errors at runtime for anything that tries to read the annotation via VersionInfo. This also causes a unit test failure in TestYarnVersionInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
