cxzl25 commented on issue #25084: [SPARK-28314][SQL] Use the same MemoryManager when building HashedRelation URL: https://github.com/apache/spark/pull/25084#issuecomment-513708465 I found some logs in the production environment. Sometimes ```BroadcastMode#transform``` can succeed after multiple failed allocation logs appear. If use ```Runtime.getRuntime.maxMemory*spark.memory.fraction``` or ```Runtime.getRuntime.maxMemory```, although the allocation(```TaskMemoryManager#allocatePage```) is unsuccessful, ```memoryForTask``` accumulates the number of allocations each time, so after several failures, it will exit the recursive ```allocatePage``` call and throw a ```SparkOutOfMemoryError```. ```ExecutionMemoryPool#acquireMemory``` https://github.com/apache/spark/blob/a950570f91db56cbae488c82def49cd0da16e996/core/src/main/scala/org/apache/spark/memory/ExecutionMemoryPool.scala#L142-L143 ```MemoryConsumer#allocateArray``` https://github.com/apache/spark/blob/e3f7ca37db0a63653f2e87f19442e8e84ced51cd/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java#L116-L123 ``` 19/07/18 20:10:25 INFO BroadcastExchangeExec: BroadcastExec numRows: 8694366, collectTime: 6027 19/07/18 20:10:31 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:32 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:32 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:33 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:33 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:34 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:35 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:35 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:35 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:36 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:36 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:37 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:38 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:38 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:39 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:40 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:40 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:41 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:41 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:41 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:42 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:42 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:42 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:43 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:43 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:44 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:44 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:45 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:45 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:45 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:46 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:46 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:46 WARN TaskMemoryManager: Failed to allocate a page (536870912 bytes), try again. 19/07/18 20:10:49 INFO BroadcastExchangeExec: BroadcastExec dataSize: 1526726656, buildTime: 24519 19/07/18 20:10:49 INFO UnifiedMemoryManager: Will not store broadcast_12 as the required space (2289042068 bytes) exceeds our memory limit (1243191705 bytes) 19/07/18 20:10:49 WARN MemoryStore: Not enough space to cache broadcast_12 in memory! (computed 1456.0 MB so far) 19/07/18 20:10:49 INFO MemoryStore: Memory use = 368.7 KB (blocks) + 1024.0 KB (scratch space shared across 1 tasks(s)) = 1392.7 KB. Storage limit = 1185.6 MB. 19/07/18 20:10:49 WARN BlockManager: Persisting block broadcast_12 to disk instead. ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
