[
https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-28208.
-----------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25006
[https://github.com/apache/spark/pull/25006]
> When upgrading to ORC 1.5.6, the reader needs to be closed.
> -----------------------------------------------------------
>
> Key: SPARK-28208
> URL: https://issues.apache.org/jira/browse/SPARK-28208
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Major
> Fix For: 3.0.0
>
>
> As part of the ORC 1.5.6 release, we optimized the common pattern of:
> {code:java}
> Reader reader = OrcFile.createReader(...);
> RecordReader rows = reader.rows(...);{code}
> which used to open one file handle in the Reader and a second one in the
> RecordReader. Users were seeing this as a regression when moving from the old
> Spark ORC reader via hive to the new native reader, because it opened twice
> as many files on the NameNode.
> In ORC 1.5.6, we changed the ORC library so that it keeps the file handle in
> the Reader until it is either closed or a RecordReader is created from it.
> This has cut down the number of file open requests on the NameNode by half in
> typical spark applications. (Hive's ORC code avoided this problem by putting
> the file footer in to the input splits, but that has other problems.)
> To get the new optimization without leaking file handles, Spark needs to be
> close the readers that aren't used to create RecordReaders.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
