Steve Loughran updated PARQUET-2151:
    Summary: Drop Hadoop 1 input stream support from parquet-hadoop   (was: 
parquet-hadoop to drop Hadoop 1 input stream support)

> Drop Hadoop 1 input stream support from parquet-hadoop 
> -------------------------------------------------------
>                 Key: PARQUET-2151
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2151
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>    Affects Versions: 1.13.0
>            Reporter: Steve Loughran
>            Priority: Minor
> Parquet uses reflection to load a hadoop2 input stream, falling back to a 
> hadoop-1 compatible client if not found.
> All hadoop 2.0.2+ releases work with H2SeekableInputStream, so 
> H1SeekableInputStream can be cut and the binding to H2SeekableInputStream 
> reworked to avoid needing reflection. This would make it a lot easier to 
> probe for/use the bytebuffer input, and line the code up for more recent 
> hadoop releases.
> One thing H1SeekableInputStream does do is read into a temp array if the 
> FSDataInputStream doesn't support , that is, doesn't implement 
> ByteBufferReadable.
> but FSDataInputStream simply forwards that to the inner stream, if it too 
> implements ByteBufferReadable. Filesystems which don't (the cloud stores) 
> can't be read through H2SeekableInputStream.read(ByteBufferReadable). If this 
> desired, H2SeekableInputStream will need to dynamically downgrade to 
> DelegatingSeekableInputStream's base methods if a call to 
> FSDataInputStream.read(ByteBuffer) fails.

This message was sent by Atlassian Jira

Reply via email to