Colm Dougan created HADOOP-20058:
------------------------------------
Summary: S3A: default analytics input stream keeps per-object
state for the life of the filesystem, so heap grows unbounded in long-running
processes
Key: HADOOP-20058
URL: https://issues.apache.org/jira/browse/HADOOP-20058
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.4.3, 3.5.0
Environment: hadoop-aws 3.5.0 (also 3.4.3), analyticsaccelerator-s3
1.3.1, default S3A settings (fs.s3a.input.stream.type unset, so analytics).
Java 17, Linux. Reproduced with hadoop-aws's mock S3 client (no endpoint) and
against S3-compatible stores (MinIO, Ceph RGW).
Reporter: Colm Dougan
Attachments: s3a-analytics-blobstore-repro.zip
Since HADOOP-19559 (3.4.3, 3.5.0), the default value of
{{fs.s3a.input.stream.type}} changed from {{classic}} to {{analytics}}. As a
result, each {{S3AFileSystem}} reads through an analyticsaccelerator-s3
{{BlobStore}}. In the library version Hadoop ships (1.3.1), that store keeps an
entry for every distinct object read until the filesystem is closed: its
periodic cleanup frees the cached data, but not the entries.
With the filesystem cache, an S3A filesystem typically lives as long as the
process, so a long-running service that keeps reading new objects grows its
heap without limit. Short-lived jobs don't notice because the filesystem is
discarded at exit.
The library issue, with a library-only reproduction, is
[https://github.com/awslabs/analytics-accelerator-s3/issues/371], and a
proposed fix is [https://github.com/awslabs/analytics-accelerator-s3/pull/372].
h3. Reproduction through S3A
The attached Gradle project reproduces it through a Hadoop 3.5.0
{{S3AFileSystem}} with default settings, using hadoop-aws's own
{{AbstractS3AMockTest}} and {{MockS3ClientFactory}}, so no endpoint is needed
({{./gradlew test}}, about 30 seconds, Java 17). Every stream is opened, read
fully and closed.
||Check||Result||
|Entries after reading 2,000 distinct .avro objects (4 KB each)|2,000|
|After re-reading 1,000 of them|still 2,000 (one entry per distinct object)|
|12 seconds later, no reads|still 2,000|
|After reading 6,000 distinct objects|6,000, no limit reached|
|Heap held with the filesystem open|about 31 KB per object read|
|Heap held after {{fs.close()}}|back to the starting level|
|Same reads with {{fs.s3a.input.stream.type=classic}}|about 80 bytes per object|
With a 160 MB heap the default stream ran out of memory after 2,000 to 3,000
distinct objects; with {{classic}} the same 50,000 reads completed.
Whether a read leaves an entry depends on how the library handles it. A plain
{{FileSystem.open()}} of a {{.avro}} or {{.parquet}} file leaves one; {{.json}}
files, {{openFile()}} with read policy {{sequential}}, and
{{fs.s3a.experimental.input.fadvise=sequential}} don't. Apache Iceberg's
{{HadoopInputFile}} uses a plain {{fs.open(path)}}, so every Avro manifest and
manifest list read through HadoopFileIO on S3A leaves an entry. That is how we
found it: a long-running Iceberg replication service ran out of a 160 MB heap
after about 1,000 table commits.
h3. Workarounds
* {{fs.s3a.input.stream.type=classic}}
* {{fs.s3a.experimental.input.fadvise=sequential}}, at the cost of the
analytics stream's format-specific handling
h3. Suggested changes in Hadoop
# Once a fixed analyticsaccelerator-s3 release is available, upgrade to it.
# Documentation: the default change isn't visible to users today.
#* The 3.4.3 and 3.5.0 release notes pages don't mention HADOOP-19559 or the
stream type change; the note is only in the JIRA's release-note field.
#* {{hadoop-aws/src/site/markdown/tools/hadoop-aws/reading.md}} in 3.5.0 still
lists {{classic}} as the default for {{fs.s3a.input.stream.type}}, while
{{StreamIntegration.DEFAULT_STREAM_TYPE}} is {{Analytics}}.
#* Until a fixed library is shipped, it may be worth noting {{classic}} for
long-running processes there.
ATTACHMENT: s3a-analytics-blobstore-repro.zip
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
