[
https://issues.apache.org/jira/browse/ORC-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796987#comment-17796987
]
dzcxzl commented on ORC-1458:
-----------------------------
After communicating with [~hexiaoqiao] offline, I realized that the length of
wrappedStream is only used when the file is completed and not in the
UnderConstruction state.
This probably reduced the RPC of our cluster by more than 2 billion getfileinfo
RPCs.
> reduce namenode getFileinfo rpc
> --------------------------------
>
> Key: ORC-1458
> URL: https://issues.apache.org/jira/browse/ORC-1458
> Project: ORC
> Issue Type: Wish
> Components: Java, Reader
> Reporter: Mingchen_Ma
> Priority: Minor
>
> In the ReaderImpl.java code, there is the following logic:
> {code:java}
> if (maxFileLength == Long. MAX_VALUE) {
> FileStatus fileStatus = fs.getFileStatus(path);
> size = fileStatus. getLen();
> modificationTime = fileStatus. getModificationTime();
> }
> {code}
> The above logic is to obtain the length of the file so as to read the footer
> of orc. But because of this, when we read the orc file on hdfs, an open
> operation will cause an additional getFileinfo rpc operation by default
> (unless we set the file length through ReaderOptions.set before the orc open).
> Because we have opened the file in ReaderImpl, can we optimize the rpc call
> of NN in the following way (in a high-load cluster, the pressure on the
> namenode can be significantly reduced):
> {code:java}
> if (maxFileLength == Long. MAX_VALUE) {
> size = (DFSInputStream)file.getWrappedStream.getFileLength();
> modificationTime = -1;
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
