[
https://issues.apache.org/jira/browse/SPARK-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Shen updated SPARK-6738:
-----------------------------
Description:
ExternalAppendOnlyMap spill 1100M data to disk:
{code}
15/04/07 16:39:48 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.5 MB to disk (12 times so far)
/data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-994b/30/temp_local_e4347165-6263-4678-9f1d-67ad4bcd8fb5
15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.3 MB to disk (13 times so far)
/data6/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-1e29/26/temp_local_76f9900b-1b3d-4cef-b3a2-6afcde14bbd9
15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1105.8 MB to disk (14 times so far)
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
15/04/07 16:39:50 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.8 MB to disk (15 times so far)
{code}
But the file size is only 1.1M.
{code}
[tdwadmin@tdw-10-215-149-231
~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
ll -h
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
-rw-r----- 1 spark users 1.1M Apr 7 16:39
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
{code}
Here is the other spilled file.
{code}
[tdwadmin@tdw-10-215-149-231
~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
ll -h
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/*
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/09:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:39
temp_local_3a568e10-3997-4d13-adf1-e8dfe4ba4727
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/18:
total 2.2M
-rw-r----- 1 spark users 1.1M Apr 7 16:41
temp_local_66c0df48-5d79-448b-8989-84ce1a5507d0
-rw-r----- 1 spark users 1.1M Apr 7 16:39
temp_local_f6870214-bfd5-47b2-b0b9-37194b55761b
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1a:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:40
temp_local_ba1712d2-0eb8-4833-9fa6-a87ee670826c
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1b:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:41
temp_local_1d1df5b7-846c-4bcd-a9de-c328e50e62db
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1f:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:41
temp_local_38c6a144-b588-49b1-b0f0-b91b31c2e85f
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/20:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:40
temp_local_d816a301-7520-4b6e-8866-bc445f04f0c9
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/24:
total 1.1M
-rw-r----- 1 spark users 1.1M Apr 7 16:40
temp_local_7a3619cf-20e1-4815-8b1f-faeefde40d73
/data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/2e:
total 2.2M
-rw-r----- 1 spark users 1.1M Apr 7 16:39
temp_local_1e366a65-ced7-4b17-a085-5f873ff6dc43
-rw-r----- 1 spark users 1.1M Apr 7 16:41
temp_local_6772b815-11ee-413a-bc6c-d0dc9cbffc51
{code}
The estimateSize is hugh difference with spill file size
was:
ExternalAppendOnlyMap spill 1100M data to disk:
{code}
15/04/07 16:39:48 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.5 MB to disk (12 times so far)
/data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-994b/30/temp_local_e4347165-6263-4678-9f1d-67ad4bcd8fb5
15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.3 MB to disk (13 times so far)
/data6/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-1e29/26/temp_local_76f9900b-1b3d-4cef-b3a2-6afcde14bbd9
15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1105.8 MB to disk (14 times so far)
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
15/04/07 16:39:50 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
in-memory map of 1106.8 MB to disk (15 times so far)
{code}
But the file size is only 1.1M.
{code}
[tdwadmin@tdw-10-215-149-231
~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
ll -h
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
-rw-r----- 1 spark users 1.1M Apr 7 16:39
/data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
{code}
The estimateSize is hugh difference with spill file size
> EstimateSize is difference with spill file size
> ------------------------------------------------
>
> Key: SPARK-6738
> URL: https://issues.apache.org/jira/browse/SPARK-6738
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Hong Shen
>
> ExternalAppendOnlyMap spill 1100M data to disk:
> {code}
> 15/04/07 16:39:48 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
> in-memory map of 1106.5 MB to disk (12 times so far)
> /data11/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-994b/30/temp_local_e4347165-6263-4678-9f1d-67ad4bcd8fb5
> 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
> in-memory map of 1106.3 MB to disk (13 times so far)
> /data6/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-1e29/26/temp_local_76f9900b-1b3d-4cef-b3a2-6afcde14bbd9
> 15/04/07 16:39:49 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
> in-memory map of 1105.8 MB to disk (14 times so far)
> /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> 15/04/07 16:39:50 INFO collection.ExternalAppendOnlyMap: Thread 51 spilling
> in-memory map of 1106.8 MB to disk (15 times so far)
> {code}
> But the file size is only 1.1M.
> {code}
> [tdwadmin@tdw-10-215-149-231
> ~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
> ll -h
> /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> -rw-r----- 1 spark users 1.1M Apr 7 16:39
> /data7/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-f883/26/temp_local_3ade0aec-ac1d-469d-bc99-b6fa87cb649b
> {code}
> Here is the other spilled file.
> {code}
> [tdwadmin@tdw-10-215-149-231
> ~/tdwenv/tdwgaia/logs/container-logs/application_1423737010718_40308573/container_1423737010718_40308573_01_000008]$
> ll -h
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/*
>
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/09:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:39
> temp_local_3a568e10-3997-4d13-adf1-e8dfe4ba4727
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/18:
> total 2.2M
> -rw-r----- 1 spark users 1.1M Apr 7 16:41
> temp_local_66c0df48-5d79-448b-8989-84ce1a5507d0
> -rw-r----- 1 spark users 1.1M Apr 7 16:39
> temp_local_f6870214-bfd5-47b2-b0b9-37194b55761b
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1a:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:40
> temp_local_ba1712d2-0eb8-4833-9fa6-a87ee670826c
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1b:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:41
> temp_local_1d1df5b7-846c-4bcd-a9de-c328e50e62db
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/1f:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:41
> temp_local_38c6a144-b588-49b1-b0f0-b91b31c2e85f
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/20:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:40
> temp_local_d816a301-7520-4b6e-8866-bc445f04f0c9
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/24:
> total 1.1M
> -rw-r----- 1 spark users 1.1M Apr 7 16:40
> temp_local_7a3619cf-20e1-4815-8b1f-faeefde40d73
> /data3/yarnenv/local/usercache/spark/appcache/application_1423737010718_40308573/spark-local-20150407163931-fe54/2e:
> total 2.2M
> -rw-r----- 1 spark users 1.1M Apr 7 16:39
> temp_local_1e366a65-ced7-4b17-a085-5f873ff6dc43
> -rw-r----- 1 spark users 1.1M Apr 7 16:41
> temp_local_6772b815-11ee-413a-bc6c-d0dc9cbffc51
> {code}
> The estimateSize is hugh difference with spill file size
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
