Hi It's what I said in the hadoop file format PR. When I discussed with Davor and Pei about the refactoring of the IOChannelFactory, I proposed to refactore hdfs IO to deal with hadoop file format on top of the file IO.
Regards JB On Feb 15, 2017, 15:13, at 15:13, Raghu Angadi <rang...@google.com.INVALID> wrote: >I skimmed through HdfsIO and I think it is essentially >HahdoopInpuFormatIO >with FileInputFormat. I would pretty much move most of the code to >HadoopInputFormatIO (just make HdfsIO a specific instance of HIF_IO). > >On Wed, Feb 15, 2017 at 9:15 AM, Dipti Kulkarni < >dipti_dkulka...@persistent.com> wrote: > >> Hello there! >> I am working on writing a Read IO for Hadoop InputFormat. This will >enable >> reading from any datasource which supports Hadoop InputFormat, i.e. >> provides source to read from InputFormat for integration with Hadoop. >> It makes sense for the HadoopInputFormatIO to share some code with >the >> HdfsIO - WritableCoder in particular, but also some helper classes >like >> SerializableSplit etc. I was wondering if we could move HDFS and >> HadoopInputFormat into a shared module for Hadoop IO in general >instead of >> maintaining them separately. >> Do let me know on what you think, please let me know if you can think >of >> any other ideas too. >> >> Thanks, >> Dipti >> >> >> DISCLAIMER >> ========== >> This e-mail may contain privileged and confidential information which >is >> the property of Persistent Systems Ltd. It is intended only for the >use of >> the individual or entity to which it is addressed. If you are not the >> intended recipient, you are not authorized to read, retain, copy, >print, >> distribute or use this message. If you have received this >communication in >> error, please notify the sender and delete all copies of this >message. >> Persistent Systems Ltd. does not accept any liability for virus >infected >> mails. >> >>
