+1 WebHDFS has been around for two years and has become mature. It is a rewrite of HFTP and is intended to replace HFTP as stated in my blog [1]. Now is indeed a good time to deprecate HFTP.
Haohui, thanks a lot for adding swebhdfs and fixing the secure setups! Tsz-Wo [1] http://hortonworks.com/blog/webhdfs-–-http-rest-access-to-hdfs/ On Tuesday, November 26, 2013 2:30 PM, Haohui Mai <h...@hortonworks.com> wrote: I meant HDP 1.3, which is based on Hadoop 1.2. > >~Haohui > > >On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <h...@hortonworks.com> wrote: > >> Hi Lohit, >> >> Some of our customers are running Hadoop 1.3, and they run distcp to >> transfer data via webhdfs in production. >> >> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As far >> as I know webhdfs is the most reliable way to transfer data between secure >> and insecure clusters across different Hadoop versions. >> >> ~Haohui >> >> >> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lohit.vijayar...@gmail.com> wrote: >> >>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in >>> production? >>> Reason I ask is because we use lot of hftp to transfer data between Hadoop >>> 1.0 and Hadoop 2.0 as of today. >>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would >>> have to move to webhdfs, right? >>> >>> >>> >>> 2013/11/26 Suresh Srinivas <sur...@hortonworks.com> >>> >>> > Thanks Haohui for all your hard work in this area. I am +1 on this >>> > proposal. >>> > >>> > >>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <h...@hortonworks.com> >>> wrote: >>> > >>> > > Hi, >>> > > >>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs / >>> swebhdfs >>> > in >>> > > various set ups. Now we have reached the state that all the above file >>> > > systems can work in both secure and insecure clusters, and transfer >>> data >>> > > through both http and https. >>> > > >>> > > Taking a step back, these file systems are very similar, and I'm >>> > wondering >>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop right >>> > now. >>> > > >>> > > The main reason is that hftp / hsftp only provides a strict subset of >>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs / >>> > swebhdfs >>> > > support writes and HA which hftp / hsftp do not support. It's more >>> > natural >>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp >>> around >>> > > seems introducing more work. >>> > > >>> > > Another reason is that webhdfs has been supported since Hadoop 1, thus >>> > > getting rid of hftp / hsftp does not seem removing any features, even >>> the >>> > > users are trying to migrate from Hadoop 1 to Hadoop 2. >>> > > >>> > > >>> > > Your ideas are appreciated. >>> > > >>> > > Thanks, >>> > > Haohui >>> > > >>> > > -- >>> > > CONFIDENTIALITY NOTICE >>> > > NOTICE: This message is intended for the use of the individual or >>> entity >>> > to >>> > > which it is addressed and may contain information that is >>> confidential, >>> > > privileged and exempt from disclosure under applicable law. If the >>> reader >>> > > of this message is not the intended recipient, you are hereby notified >>> > that >>> > > any printing, copying, dissemination, distribution, disclosure or >>> > > forwarding of this communication is strictly prohibited. If you have >>> > > received this communication in error, please contact the sender >>> > immediately >>> > > and delete it from your system. Thank You. >>> > > >>> > >>> > >>> > >>> > -- >>> > http://hortonworks.com/download/ >>> > >>> > -- >>> > CONFIDENTIALITY NOTICE >>> > NOTICE: This message is intended for the use of the individual or >>> entity to >>> > which it is addressed and may contain information that is confidential, >>> > privileged and exempt from disclosure under applicable law. If the >>> reader >>> > of this message is not the intended recipient, you are hereby notified >>> that >>> > any printing, copying, dissemination, distribution, disclosure or >>> > forwarding of this communication is strictly prohibited. If you have >>> > received this communication in error, please contact the sender >>> immediately >>> > and delete it from your system. Thank You. >>> > >>> >>> >>> >>> -- >>> Have a Nice Day! >>> Lohit >>> >> >> > >-- >CONFIDENTIALITY NOTICE >NOTICE: This message is intended for the use of the individual or entity to >which it is addressed and may contain information that is confidential, >privileged and exempt from disclosure under applicable law. If the reader >of this message is not the intended recipient, you are hereby notified that >any printing, copying, dissemination, distribution, disclosure or >forwarding of this communication is strictly prohibited. If you have >received this communication in error, please contact the sender immediately >and delete it from your system. Thank You. > > >
