[ https://issues.apache.org/jira/browse/SPARK-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-22062: ------------------------------------ Assignee: Apache Spark > BlockManager does not account for memory consumed by remote fetches > ------------------------------------------------------------------- > > Key: SPARK-22062 > URL: https://issues.apache.org/jira/browse/SPARK-22062 > Project: Spark > Issue Type: Bug > Components: Block Manager > Affects Versions: 2.2.0 > Reporter: Sergei Lebedev > Assignee: Apache Spark > Priority: Minor > > We use Spark exclusively with {{StorageLevel.DiskOnly}} as our workloads are > very sensitive to memory usage. Recently, we've spotted that the jobs > sometimes OOM leaving lots of byte[] arrays on the heap. Upon further > investigation, we've found that the arrays come from > {{BlockManager.getRemoteBytes}}, which > [calls|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L638] > {{BlockTransferService.fetchBlockSync}}, which in its turn would > [allocate|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/BlockTransferService.scala#L99] > an on-heap {{ByteBuffer}} of the same size as the block (e.g. full > partition), if the block was successfully retrieved over the network. > This memory is not accounted towards Spark storage/execution memory and could > potentially lead to OOM if {{BlockManager}} fetches too many partitions in > parallel. I wonder if this is intentional behaviour, or in fact a bug? -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org