After this is merged in is Windows still going to be a second class citizen but happens to work for more than just development or is it a fully supported platform where if something breaks it can block a release? How do we as a community intend to keep Windows support from breaking? We don't have any Jenkins slaves to be able to run nightly tests to validate everything still compiles/runs. This is not a blocker for me because we often rely on individuals and groups to test Hadoop, but I do think we need to have this discussion before we put it in.
--Bobby On 2/26/13 4:55 PM, "Suresh Srinivas" <sur...@hortonworks.com> wrote: >I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. >I >am happy to announce that we are ready for the merge. > >Here is a brief recap on the highlights of the work done: >- Command-line scripts for the Hadoop surface area >- Mapping the HDFS permissions model to Windows >- Abstracted and reconciled mismatches around differences in Path >semantics >in Java and Windows >- Native Task Controller for Windows >- Implementation of a Block Placement Policy to support cloud >environments, >more specifically Azure. >- Implementation of Hadoop native libraries for Windows (compression >codecs, native I/O) >- Several reliability issues, including race-conditions, intermittent test >failures, resource leaks. >- Several new unit test cases written for the above changes > >Please find the details of the work in CHANGES.branch-trunk-win.txt - >Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>, >and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work >ported from branch-1-win to a branch based on trunk. > >For details of the testing done, please see the thread - >http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562< >https://issues.apache.org/jira/browse/HADOOP-8562>. > >This was a large undertaking that involved developing code, testing the >entire Hadoop stack, including scale tests. This is made possible only >with >the contribution from many many folks in the community. Following people >contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha, >Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur >Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas >Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya >Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh >Srinivas and Sanjay Radia. There are many others who contributed as well >providing feedback and comments on numerous jiras. > >The vote will run for seven days and will end on March 5, 6:00PM PST. > >Regards, >Suresh > > > > >On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman ><mah...@microsoft.com>wrote: > >> It is super exciting to look at the prospect of these changes being >>merged >> to trunk. Having Windows as one of the supported Hadoop platforms is a >> fantastic opportunity both for the Hadoop project and Microsoft >>customers. >> >> This work began around a year back when a few of us started with a basic >> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have >> made significant progress in the following areas: >> (PS: Some of these items are already included in Suresh's email, but >> including again for completeness) >> >> - Command-line scripts for the Hadoop surface area >> - Mapping the HDFS permissions model to Windows >> - Abstracted and reconciled mismatches around differences in Path >> semantics in Java and Windows >> - Native Task Controller for Windows >> - Implementation of a Block Placement Policy to support cloud >> environments, more specifically Azure. >> - Implementation of Hadoop native libraries for Windows (compression >> codecs, native I/O) - Several reliability issues, including >> race-conditions, intermittent test failures, resource leaks. >> - Several new unit test cases written for the above changes >> >> In the process, we have closely engaged with the Apache open source >> community and have got great support and assistance from the community >>in >> terms of contributing fixes, code review comments and commits. >> >> In addition, the Hadoop team at Microsoft has also made good progress in >> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many >>of >> these changes have already been committed to the respective trunks with >> help from various committers and contributors. It is great to see the >> commitment of the community to support multiple platforms, and we look >> forward to the day when a developer/customer is able to successfully >>deploy >> a complete solution stack based on Apache Hadoop releases. >> >> Next Steps: >> >> All of the above changes are part of the Windows Azure HDInsight and >> HDInsight Server products from Microsoft. We have successfully >>on-boarded >> several internal customers and have been running production workloads on >> Windows Azure HDInsight. Our vision is to create a big data platform >>based >> on Hadoop, and we are committed to helping make Hadoop a world-class >> solution that anyone can use to solve their biggest data challenges. >> >> As an immediate next step, we would like to have a discussion around how >> we can ensure that the quality of the mainline Hadoop branches on >>Windows >> is maintained. To this end, we would like to get to the state where we >>have >> pre-checkin validation gates and nightly test runs enabled on Windows. >>If >> you have any suggestions around this, please do send an email. We are >> committed to helping sustain the long-term quality of Hadoop on both >>Linux >> and Windows. >> >> We sincerely thank the community for their contribution and support so >> far. And hope to continue having a close engagement in the future. >> >> -Microsoft HDInsight Team >> >> >> -----Original Message----- >> From: Suresh Srinivas [mailto:sur...@hortonworks.com] >> Sent: Thursday, February 7, 2013 5:42 PM >> To: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; >> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org >> Subject: Heads up - merge branch-trunk-win to trunk >> >> The support for Hadoop on Windows was proposed in HADOOP-8079< >> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago. >>The >> goal was to make Hadoop natively integrated, full-featured, and >>performance >> and scalability tuned on Windows Server or Windows Azure. >> We are happy to announce that a lot of progress has been made in this >> regard. >> >> Initial work started in a feature branch, branch-1-win, based on >>branch-1. >> The details related to the work done in the branch can be seen in >> CHANGES.txt< >> >>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES. >>branch-1-win.txt?view=markup >> >. >> This work has been ported to a branch, branch-trunk-win, based on trunk. >> Merge patch for this is available on >> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562> >> . >> >> Highlights of the work done so far: >> 1. Necessary changes in Hadoop to run natively on Windows. These changes >> handle differences in platforms related to path names, process/task >> management etc. >> 2. Addition of winutils tools for managing file permissions and >>ownership, >> user group mapping, hardlinks, symbolic links, chmod, disk utilization, >>and >> process/task management. >> 3. Added cmd scripts equivalent to existing shell scripts >> hadoop-daemon.sh, start and stop scripts. >> 4. Addition of block placement policy implemnation to support cloud >> enviroment, more specifically Azure. >> >> We are very close to wrapping up the work in branch-trunk-win and >>getting >> ready for a merge. Currently the merge patch is passing close to 100% of >> unit tests on Linux. Soon I will call for a vote to merge this branch >>into >> trunk. >> >> Next steps: >> 1. Call for vote to merge branch-trunk-win to trunk, when the work >> completes and precommit build is clean. >> 2. Start a discussion on adding Jenkins precommit builds on windows and >> how to integrate that with the existing commit process. >> >> Let me know if you have any questions. >> >> Regards, >> Suresh >> >> > > >-- >http://hortonworks.com/download/
