[
https://issues.apache.org/jira/browse/AVRO-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved AVRO-1244.
---------------------------
Resolution: Invalid
I missed {{FsInput}} in avro-mapred. It already implements this! :)
> Provide a SeekableInput implementation for FileSystem retrieved output streams
> ------------------------------------------------------------------------------
>
> Key: AVRO-1244
> URL: https://issues.apache.org/jira/browse/AVRO-1244
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Harsh J
> Priority: Minor
>
> To use the DFW#appendTo API, one needs to pass a SeekableInput interface
> object. Avro provides a usable utility for files that can be represented by a
> File object, but in the Hadoop land, HDFS and other FSes can't be represented
> via a File object and need a longer route to implement this interface.
> We can add a simple HadoopSeekableFSInput or so that can take Hadoop provided
> objects and wrap it into a SeekableInput interface ready for passing to Avro.
> I propose something of the following type:
> {code}
> public static class HadoopSeekableFSInput implements SeekableInput {
> FSDataInputStream in;
> long length;
>
> public SeekableFSInput(FSDataInputStream in, long length) {
> this.in = in;
> this.length = length;
> }
>
> public void close() throws IOException {
> in.close();
> }
>
> public void seek(long p) throws IOException {
> in.seek(p);
> }
>
> public long tell() throws IOException {
> return in.getPos();
> }
>
> public long length() throws IOException {
> return length;
> }
>
> public int read(byte[] b, int off, int len) throws IOException {
> return in.read(b, off, len);
> }
> }
> {code}
> The above can be constructed by users via a simple call such as {{new
> HadoopSeekableFSInput(fs.open(filePath),
> fs.getFileStatus(filePath).getLen())}}.
> Ideally this class should belong in the avro core module but that strictly
> does not depend on Hadoop-Common today, and hence somewhere else may be more
> suitable.
> This lets users write Avro-append code such as
> https://gist.github.com/QwertyManiac/4724582 more easily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
