[
https://issues.apache.org/jira/browse/IMPALA-7556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623819#comment-16623819
]
ASF subversion and git services commented on IMPALA-7556:
---------------------------------------------------------
Commit d7b8275ed04581c4b9bef60ae0abdb7201661618 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d7b8275 ]
IMPALA-7556: part 1: handle different file systems via polymorphism
This commit reorganizes some parts of the ScanRange class.
File operations are handled through the abstract FileReader
class. The interface supports positional read that will
be needed by IMPALA-5843. The concrete file operations are
implemented in sub-classes LocalFileReader and
HdfsFileReader.
File reader classes are responsible for setting counters and
metrics related to file operations.
The core logic haven't been changed significantly, but quite
a lot code fragments were relocated.
Testing: Debug exhaustive tests passed
Change-Id: Ia3d3d2d774075008285230606b992603d5be1a82
Reviewed-on: http://gerrit.cloudera.org:8080/11444
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Clean up ScanRange
> ------------------
>
> Key: IMPALA-7556
> URL: https://issues.apache.org/jira/browse/IMPALA-7556
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
>
> For IMPALA-7543 I want to add some additional functionality to scan ranges.
> However, the code of the ScanRange class is already quite messy. It handles
> different types of files, does some buffer management, updates all kinds of
> counters.
> So, instead of complicating the code further, let's refactor the ScanRange
> class a bit.
> * Do the file operations in separate classes
> ** A new, abstract class could be invented to provide an API for file
> operations, i.e. Open(), ReadFromPos(), Close(), etc.
> *** Keep in mind that the interface must be a good fit for IMPALA-7543, i.e.
> we need positional reads from files
> ** Operations for local files and HDFS files could be implemented in child
> classes
> * Buffer management
> ** A new BufferStore class could be created
> ** This new class would be responsible for managing the unused buffers
> *** if possible, it would also handle the client and cached buffers as well
> * Counters and metrics would be updated by the corresponding new classes
> ** E.g. ImpaladMetrics::IO_MGR_NUM_OPEN_FILES would be updated by the file
> handling classes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
