Neville Li created BEAM-10787:
---------------------------------
Summary: External sorter 2048MB memory limit should be Hadoop only
Key: BEAM-10787
URL: https://issues.apache.org/jira/browse/BEAM-10787
Project: Beam
Issue Type: Bug
Components: extensions-java-sorter
Affects Versions: 2.23.0
Reporter: Neville Li
Assignee: Neville Li
The 2048MB limit only applies when using Hadoop sequence file for external sort.
It doesn't apply when sorting is entirely in memory since we're using a
{{List<KV<byte[], byte[]>>}} to buffer data and {{long}} to keep track of total
bytes. Same with native external sorter.
https://github.com/apache/beam/pull/12659
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
