Haram Lee created SPARK-61742:
---------------------------------
Summary: Open event logs with the sequential read policy
Key: SPARK-61742
URL: https://issues.apache.org/jira/browse/SPARK-61742
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Haram Lee
{{EventLogFileReader.openEventLog}} opens event logs with
{{FileSystem.open()}}. Event logs are always read from start to end, but this
call does not tell the filesystem about that.
Since HADOOP-19559 (Hadoop 3.4.3 and 3.5.0), the default S3A input stream is
the analytics stream. It keeps per-object state for the lifetime of the
filesystem, and only removes an object's state when the stream is closed if the
stream was opened with a sequential read policy (see HADOOP-20058). A History
Server that reads event logs from S3 keeps one cached FileSystem for its whole
life, so its heap grows with every event log it reads, until it runs out of
memory.
A local measurement against an S3-compatible server, reading 4,000 small event
logs with hadoop-aws 3.5.0 and the default stream type:
* {{FileSystem.open()}}: live heap grew by about 34 KB per log, and 144,100
{{MeanStatistic}} objects were retained.
* {{FileSystem.openFile()}} with {{fs.option.openfile.read.policy=sequential}}:
live heap did not grow.
Proposed fix: open event logs with {{FileSystem.openFile()}} and set
{{fs.option.openfile.read.policy}} to {{sequential}}, the same way
{{HadoopLineRecordReader}} already uses {{openFile()}} (SPARK-52482).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]