[
https://issues.apache.org/jira/browse/SPARK-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-11776.
-------------------------------
Resolution: Invalid
I don't think this describes a valid problem. For example, there is no
indication that you even have enough data to fill the cache.
> Cache fraction hardly reach 100%
> --------------------------------
>
> Key: SPARK-11776
> URL: https://issues.apache.org/jira/browse/SPARK-11776
> Project: Spark
> Issue Type: Bug
> Reporter: Wenmin Wu
>
> Even with plenty partitions as well as executor memory the cache fraction can
> hardly reach 100%. What even worse is that those uncached executor has too
> much records. Here is the summary of the aggregated metrics:
> 1 hd268dg.prod.mediav.com:19344 23 s 1 0 1 339.5
> MB / 314 5.7 MB / 1
> 10 hd268dg.prod.mediav.com:2342 23 s 1 0 1 340.0
> MB / 315 5.7 MB / 1
> 11 hd258dg.prod.mediav.com:63130 23 s 1 0 1 340.1
> MB / 315 5.7 MB / 1
> 12 hd278dg.prod.mediav.com:57601 15 s 1 0 1 340.6
> MB / 315 5.7 MB / 1
> 13 hd268dg.prod.mediav.com:38740 25 s 1 0 1 340.0
> MB / 315 5.7 MB / 1
> 14 hd258dg.prod.mediav.com:34498 22 s 1 0 1 339.8
> MB / 315 5.7 MB / 1
> 16 hd268dg.prod.mediav.com:27714 25 s 1 0 1 341.1
> MB / 316 5.7 MB / 1
> 17 hd258dg.prod.mediav.com:36584 22 s 1 0 1 340.7
> MB / 315 5.7 MB / 1
> 18 hd278dg.prod.mediav.com:37627 16 s 1 0 1 340.8
> MB / 315 5.7 MB / 1
> 19 hd268dg.prod.mediav.com:5938 25 s 1 0 1 339.8
> MB / 315 5.7 MB / 1
> 2 hd258dg.prod.mediav.com:46592 2.3 min 1 0 1
> 190.8 MB / 628778 6.5 MB / 1
> 20 hd258dg.prod.mediav.com:29043 2.2 min 1 0 1
> 190.5 MB / 627972 6.5 MB / 1
> You can see the records in executor 2 and executor 20 is too much with
> respect to the storage. I think some bugs must exists during caching stage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
