[
https://issues.apache.org/jira/browse/HADOOP-20058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colm Dougan updated HADOOP-20058:
---------------------------------
Description:
Since HADOOP-19559 (3.4.3, 3.5.0), the default value of
{{fs.s3a.input.stream.type}} changed from {{classic}} to {{analytics}}. As a
result, each {{S3AFileSystem}} reads through an analyticsaccelerator-s3
{{BlobStore}}. In the library version Hadoop ships (1.3.1), that store keeps an
entry for every distinct object read until the filesystem is closed: its
periodic cleanup frees the cached data, but not the entries.
With the filesystem cache, an S3A filesystem typically lives as long as the
process, so a long-running service that keeps reading new objects grows its
heap without limit. Short-lived jobs don't notice because the filesystem is
discarded at exit.
The library issue, with a library-only reproduction, is
[https://github.com/awslabs/analytics-accelerator-s3/issues/371], and a
proposed fix is [https://github.com/awslabs/analytics-accelerator-s3/pull/372].
h3. Which reads are affected
Not every read leaves an entry. The library chooses how to handle each stream
from the file name and the read policy, and streams it handles as sequential
remove their entry when closed. So it depends on the file type and how the file
is opened:
||Read||Entry left after close||
|{{.avro}}, plain {{FileSystem.open()}}|yes|
|{{.parquet}}, plain {{FileSystem.open()}}|yes|
|{{.json}} (also {{.csv}}, {{.txt}}), plain {{FileSystem.open()}}|no|
|{{.avro}}, {{openFile()}} with read policy {{sequential}} or {{whole-file}}|no|
|{{.avro}}, with {{fs.s3a.experimental.input.fadvise=sequential}}|no|
The attached project tests each row; {{.csv}}, {{.txt}} and {{whole-file}} are
from the library's {{ObjectFormatSelector}} and {{AnalyticsStream}}'s
read-policy mapping rather than tested.
So a plain {{open()}} of a non-text file builds up, which is the common case.
For example, Apache Iceberg's {{HadoopInputFile}} uses a plain
{{fs.open(path)}}, so every Avro manifest and manifest list read through
HadoopFileIO on S3A leaves an entry, while its {{.metadata.json}} files don't.
That is how we found it: a long-running Iceberg replication service ran out of
a 160 MB heap after about 1,000 table commits.
h3. Reproduction through S3A
The attached Gradle project reproduces it through a Hadoop 3.5.0
{{S3AFileSystem}} with default settings, using hadoop-aws's own
{{AbstractS3AMockTest}} and {{MockS3ClientFactory}}, so no endpoint is needed
({{./gradlew test}}, about 30 seconds, Java 17). Every stream is opened, read
fully and closed.
||Check||Result||
|Entries after reading 2,000 distinct .avro objects (4 KB each)|2,000|
|After re-reading 1,000 of them|still 2,000 (one entry per distinct object)|
|12 seconds later, no reads|still 2,000|
|After reading 6,000 distinct objects|6,000, no limit reached|
|Heap held with the filesystem open|about 31 KB per object read|
|Heap held after {{fs.close()}}|back to the starting level|
|Same reads with {{fs.s3a.input.stream.type=classic}}|about 80 bytes per object|
With a 160 MB heap the default stream ran out of memory after 2,000 to 3,000
distinct objects; with {{classic}} the same 50,000 reads completed.
h3. Workarounds
* {{fs.s3a.input.stream.type=classic}}
* {{fs.s3a.experimental.input.fadvise=sequential}}, at the cost of the
analytics stream's format-specific handling
h3. Suggested changes in Hadoop
# Once a fixed analyticsaccelerator-s3 release is available, upgrade to it.
# Documentation: the default change isn't visible to users today.
#* The 3.4.3 and 3.5.0 release notes pages don't mention HADOOP-19559 or the
stream type change; the note is only in the JIRA's release-note field.
#* {{hadoop-aws/src/site/markdown/tools/hadoop-aws/reading.md}} in 3.5.0 still
lists {{classic}} as the default for {{fs.s3a.input.stream.type}}, while
{{StreamIntegration.DEFAULT_STREAM_TYPE}} is {{Analytics}}.
#* Until a fixed library is shipped, it may be worth noting {{classic}} for
long-running processes there.
ATTACHMENT: [^s3a-analytics-blobstore-repro.zip]
was:
Since HADOOP-19559 (3.4.3, 3.5.0), the default value of
{{fs.s3a.input.stream.type}} changed from {{classic}} to {{analytics}}. As a
result, each {{S3AFileSystem}} reads through an analyticsaccelerator-s3
{{BlobStore}}. In the library version Hadoop ships (1.3.1), that store keeps an
entry for every distinct object read until the filesystem is closed: its
periodic cleanup frees the cached data, but not the entries.
With the filesystem cache, an S3A filesystem typically lives as long as the
process, so a long-running service that keeps reading new objects grows its
heap without limit. Short-lived jobs don't notice because the filesystem is
discarded at exit.
The library issue, with a library-only reproduction, is
[https://github.com/awslabs/analytics-accelerator-s3/issues/371], and a
proposed fix is [https://github.com/awslabs/analytics-accelerator-s3/pull/372].
h3. Reproduction through S3A
The attached Gradle project reproduces it through a Hadoop 3.5.0
{{S3AFileSystem}} with default settings, using hadoop-aws's own
{{AbstractS3AMockTest}} and {{MockS3ClientFactory}}, so no endpoint is needed
({{./gradlew test}}, about 30 seconds, Java 17). Every stream is opened, read
fully and closed.
||Check||Result||
|Entries after reading 2,000 distinct .avro objects (4 KB each)|2,000|
|After re-reading 1,000 of them|still 2,000 (one entry per distinct object)|
|12 seconds later, no reads|still 2,000|
|After reading 6,000 distinct objects|6,000, no limit reached|
|Heap held with the filesystem open|about 31 KB per object read|
|Heap held after {{fs.close()}}|back to the starting level|
|Same reads with {{fs.s3a.input.stream.type=classic}}|about 80 bytes per object|
With a 160 MB heap the default stream ran out of memory after 2,000 to 3,000
distinct objects; with {{classic}} the same 50,000 reads completed.
Whether a read leaves an entry depends on how the library handles it. A plain
{{FileSystem.open()}} of a {{.avro}} or {{.parquet}} file leaves one; {{.json}}
files, {{openFile()}} with read policy {{sequential}}, and
{{fs.s3a.experimental.input.fadvise=sequential}} don't. Apache Iceberg's
{{HadoopInputFile}} uses a plain {{fs.open(path)}}, so every Avro manifest and
manifest list read through HadoopFileIO on S3A leaves an entry. That is how we
found it: a long-running Iceberg replication service ran out of a 160 MB heap
after about 1,000 table commits.
h3. Workarounds
* {{fs.s3a.input.stream.type=classic}}
* {{fs.s3a.experimental.input.fadvise=sequential}}, at the cost of the
analytics stream's format-specific handling
h3. Suggested changes in Hadoop
# Once a fixed analyticsaccelerator-s3 release is available, upgrade to it.
# Documentation: the default change isn't visible to users today.
#* The 3.4.3 and 3.5.0 release notes pages don't mention HADOOP-19559 or the
stream type change; the note is only in the JIRA's release-note field.
#* {{hadoop-aws/src/site/markdown/tools/hadoop-aws/reading.md}} in 3.5.0 still
lists {{classic}} as the default for {{fs.s3a.input.stream.type}}, while
{{StreamIntegration.DEFAULT_STREAM_TYPE}} is {{Analytics}}.
#* Until a fixed library is shipped, it may be worth noting {{classic}} for
long-running processes there.
ATTACHMENT: s3a-analytics-blobstore-repro.zip
> S3A: default analytics input stream keeps per-object state for the life of
> the filesystem, so heap grows unbounded in long-running processes
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20058
> URL: https://issues.apache.org/jira/browse/HADOOP-20058
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.5.0, 3.4.3
> Environment: hadoop-aws 3.5.0 (also 3.4.3), analyticsaccelerator-s3
> 1.3.1, default S3A settings (fs.s3a.input.stream.type unset, so analytics).
> Java 17, Linux. Reproduced with hadoop-aws's mock S3 client (no endpoint) and
> against S3-compatible stores (MinIO, Ceph RGW).
> Reporter: Colm Dougan
> Priority: Major
> Attachments: s3a-analytics-blobstore-repro.zip
>
>
> Since HADOOP-19559 (3.4.3, 3.5.0), the default value of
> {{fs.s3a.input.stream.type}} changed from {{classic}} to {{analytics}}. As a
> result, each {{S3AFileSystem}} reads through an analyticsaccelerator-s3
> {{BlobStore}}. In the library version Hadoop ships (1.3.1), that store keeps
> an entry for every distinct object read until the filesystem is closed: its
> periodic cleanup frees the cached data, but not the entries.
> With the filesystem cache, an S3A filesystem typically lives as long as the
> process, so a long-running service that keeps reading new objects grows its
> heap without limit. Short-lived jobs don't notice because the filesystem is
> discarded at exit.
> The library issue, with a library-only reproduction, is
> [https://github.com/awslabs/analytics-accelerator-s3/issues/371], and a
> proposed fix is
> [https://github.com/awslabs/analytics-accelerator-s3/pull/372].
> h3. Which reads are affected
> Not every read leaves an entry. The library chooses how to handle each stream
> from the file name and the read policy, and streams it handles as sequential
> remove their entry when closed. So it depends on the file type and how the
> file is opened:
> ||Read||Entry left after close||
> |{{.avro}}, plain {{FileSystem.open()}}|yes|
> |{{.parquet}}, plain {{FileSystem.open()}}|yes|
> |{{.json}} (also {{.csv}}, {{.txt}}), plain {{FileSystem.open()}}|no|
> |{{.avro}}, {{openFile()}} with read policy {{sequential}} or
> {{whole-file}}|no|
> |{{.avro}}, with {{fs.s3a.experimental.input.fadvise=sequential}}|no|
> The attached project tests each row; {{.csv}}, {{.txt}} and {{whole-file}}
> are from the library's {{ObjectFormatSelector}} and {{AnalyticsStream}}'s
> read-policy mapping rather than tested.
> So a plain {{open()}} of a non-text file builds up, which is the common case.
> For example, Apache Iceberg's {{HadoopInputFile}} uses a plain
> {{fs.open(path)}}, so every Avro manifest and manifest list read through
> HadoopFileIO on S3A leaves an entry, while its {{.metadata.json}} files
> don't. That is how we found it: a long-running Iceberg replication service
> ran out of a 160 MB heap after about 1,000 table commits.
> h3. Reproduction through S3A
> The attached Gradle project reproduces it through a Hadoop 3.5.0
> {{S3AFileSystem}} with default settings, using hadoop-aws's own
> {{AbstractS3AMockTest}} and {{MockS3ClientFactory}}, so no endpoint is needed
> ({{./gradlew test}}, about 30 seconds, Java 17). Every stream is opened, read
> fully and closed.
> ||Check||Result||
> |Entries after reading 2,000 distinct .avro objects (4 KB each)|2,000|
> |After re-reading 1,000 of them|still 2,000 (one entry per distinct object)|
> |12 seconds later, no reads|still 2,000|
> |After reading 6,000 distinct objects|6,000, no limit reached|
> |Heap held with the filesystem open|about 31 KB per object read|
> |Heap held after {{fs.close()}}|back to the starting level|
> |Same reads with {{fs.s3a.input.stream.type=classic}}|about 80 bytes per
> object|
> With a 160 MB heap the default stream ran out of memory after 2,000 to 3,000
> distinct objects; with {{classic}} the same 50,000 reads completed.
> h3. Workarounds
> * {{fs.s3a.input.stream.type=classic}}
> * {{fs.s3a.experimental.input.fadvise=sequential}}, at the cost of the
> analytics stream's format-specific handling
> h3. Suggested changes in Hadoop
> # Once a fixed analyticsaccelerator-s3 release is available, upgrade to it.
> # Documentation: the default change isn't visible to users today.
> #* The 3.4.3 and 3.5.0 release notes pages don't mention HADOOP-19559 or the
> stream type change; the note is only in the JIRA's release-note field.
> #* {{hadoop-aws/src/site/markdown/tools/hadoop-aws/reading.md}} in 3.5.0
> still lists {{classic}} as the default for {{fs.s3a.input.stream.type}},
> while {{StreamIntegration.DEFAULT_STREAM_TYPE}} is {{Analytics}}.
> #* Until a fixed library is shipped, it may be worth noting {{classic}} for
> long-running processes there.
> ATTACHMENT: [^s3a-analytics-blobstore-repro.zip]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
