Yes, FileSystem "plugins" will use a scheme. Other connectors will use (as it's
already the case) DoFn/Source transforms.
Regards
JB
On 04/11/2017 05:05 PM, Shen Li wrote:
Hi JB,
Thanks a lot for your response. Does it mean all file-based IO will be
added as schemes using IOChannelFactory (or the new name FileSystem). All
others, e.g., HTTP, TCP, KV-store, DB, message-queue, should be source/sink
transforms?
Thanks,
Shen
On Tue, Apr 11, 2017 at 10:29 AM, Jean-Baptiste Onofré <[email protected]>
wrote:
Hi Shen,
We are doing a refactoring of the file IO (IOChannelFactory). Thanks to
this refactoring, you will be able to use a scheme for hdfs (or s3, ...)
with different format (avro, text, hadoop input format, ...).
It means that HdfsIO will be deprecated (to be removed at some point). I'm
working on couple of PRs to leverage the new file IO layer.
Regards
JB
On 04/11/2017 03:56 PM, Shen Li wrote:
Hi,
Is there any reason why HDFS IO is implemented as a BoundedSource while
Google Cloud Storage is implemented as a scheme ("gs://") for TextIO? To
contribute a new IO connector, how can I determine whether it should be
implemented as a source transform or as a scheme for the TextIO?
Thanks,
Shen
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
