Sergei Lebedev created SPARK-22062: -------------------------------------- Summary: BlockManager does not account for memory consumed by remote fetches Key: SPARK-22062 URL: https://issues.apache.org/jira/browse/SPARK-22062 Project: Spark Issue Type: Bug Components: Block Manager Affects Versions: 2.2.0 Reporter: Sergei Lebedev Priority: Minor
We use Spark exclusively with {{StorageLevel.DiskOnly}} as our workloads are very sensitive to memory usage. Recently, we've spotted that the jobs sometimes OOM leaving lots of byte[] arrays on the heap. Upon further investigation, we've found that the arrays come from {{BlockManager.getRemoteBytes}}, which [calls|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L638] {{BlockTransferService.fetchBlockSync}}, which in its turn would [allocate|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/BlockTransferService.scala#L99] an on-heap {{ByteBuffer}} of the same size as the block (e.g. full partition), if the block was successfully retrieved over the network. This memory is not accounted towards Spark storage/execution memory and could potentially lead to OOM if {{BlockManager}} fetches too many partitions in parallel. I wonder if this is intentional behaviour, or in fact a bug? -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org