Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

I don't think there is any conflict here. On Tue, Dec 13, 2016 at 12:34 PM, Pei He wrote: > One design decision made during previous design discussion [1] is > "Replacing > FilePath with URI for resolving files paths". This has been brought back to > dev@ mailing list

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

How can I unsubscribe? I will be away from this subject for sometime Will rejoin once I get back Thanks colleagues Happy holidays Sent from my iPhone > On Dec 13, 2016, at 12:34 PM, Pei He wrote: > > One design decision made during previous design discussion [1] is

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

One design decision made during previous design discussion [1] is "Replacing FilePath with URI for resolving files paths". This has been brought back to dev@ mailing list in my previous email. Comment [2] asked me to clarify the impact on Windows OS users because users have to specify the path in

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

Thanks for the thorough answers. It all sounds good to me. On Tue, Dec 6, 2016 at 12:57 PM, Pei He wrote: > Thanks Kenn for the feedback and questions. > > I responded inline. > > On Mon, Dec 5, 2016 at 7:49 PM, Kenneth Knowles > wrote: > > >

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

Thanks Kenn for the feedback and questions. I responded inline. On Mon, Dec 5, 2016 at 7:49 PM, Kenneth Knowles wrote: > I really like this document. It is easy to read and informative. Three > things not addressed by the document: > > 1. Major Beam use cases. I'm sure

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

Thanks JB for the feedback. Yes, we should provide a hadoop.fs.FileSystem adaptor. As you said, it will make a range of file system available in Beam. And, people can choose to implement BeamFileSystem directly to get the best performance (For example, providing bulk operations.) -- Pei On

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

Hi Pei, rethinking about that, I understand that the purpose of the Beam filesystem is to avoid to bring a bunch of dependencies into the core. That makes perfect sense. So, I agree that a Beam filesystem abstract is fine. My point is that we should provide a HadoopFilesystem

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

Hi JB, My proposals are based on the current IOChannelFactory, and how they are used in FileBasedSink. Let's me spend more time to investigate Hadoop FileSystem interface. -- Pei On Thu, Nov 17, 2016 at 1:21 AM, Jean-Baptiste Onofré wrote: > By the way, Pei, for the record:

Re: [PROPOSAL] "IOChannelFactory" Redesign and Make it Configurable

By the way, Pei, for the record: why introducing BeamFileSystem and not using the Hadoop FileSystem interface ? Thanks Regards JB On 11/17/2016 01:09 AM, Pei He wrote: Hi, I am working on BEAM-59 "IOChannelFactory redesign". The goals are: 1.