[ https://issues.apache.org/jira/browse/SPARK-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenmin Wu updated SPARK-11776: ------------------------------ Description: Even with plenty partitions as well as executor memory the cache fraction can hardly reach 100%. What even worse is that those uncached executor has too much records. Here is the summary of the aggregated metrics: 1 hd268dg.prod.mediav.com:19344 23 s 1 0 1 339.5 MB / 314 5.7 MB / 1 10 hd268dg.prod.mediav.com:2342 23 s 1 0 1 340.0 MB / 315 5.7 MB / 1 11 hd258dg.prod.mediav.com:63130 23 s 1 0 1 340.1 MB / 315 5.7 MB / 1 12 hd278dg.prod.mediav.com:57601 15 s 1 0 1 340.6 MB / 315 5.7 MB / 1 13 hd268dg.prod.mediav.com:38740 25 s 1 0 1 340.0 MB / 315 5.7 MB / 1 14 hd258dg.prod.mediav.com:34498 22 s 1 0 1 339.8 MB / 315 5.7 MB / 1 16 hd268dg.prod.mediav.com:27714 25 s 1 0 1 341.1 MB / 316 5.7 MB / 1 17 hd258dg.prod.mediav.com:36584 22 s 1 0 1 340.7 MB / 315 5.7 MB / 1 18 hd278dg.prod.mediav.com:37627 16 s 1 0 1 340.8 MB / 315 5.7 MB / 1 19 hd268dg.prod.mediav.com:5938 25 s 1 0 1 339.8 MB / 315 5.7 MB / 1 2 hd258dg.prod.mediav.com:46592 2.3 min 1 0 1 190.8 MB / 628778 6.5 MB / 1 20 hd258dg.prod.mediav.com:29043 2.2 min 1 0 1 190.5 MB / 627972 6.5 MB / 1 You can see the records in executor 2 and executor 20 is too much with respect to the storage. I think some bugs must exists during caching stage. > Cache fraction hardly reach 100% > -------------------------------- > > Key: SPARK-11776 > URL: https://issues.apache.org/jira/browse/SPARK-11776 > Project: Spark > Issue Type: Bug > Reporter: Wenmin Wu > > Even with plenty partitions as well as executor memory the cache fraction can > hardly reach 100%. What even worse is that those uncached executor has too > much records. Here is the summary of the aggregated metrics: > 1 hd268dg.prod.mediav.com:19344 23 s 1 0 1 339.5 > MB / 314 5.7 MB / 1 > 10 hd268dg.prod.mediav.com:2342 23 s 1 0 1 340.0 > MB / 315 5.7 MB / 1 > 11 hd258dg.prod.mediav.com:63130 23 s 1 0 1 340.1 > MB / 315 5.7 MB / 1 > 12 hd278dg.prod.mediav.com:57601 15 s 1 0 1 340.6 > MB / 315 5.7 MB / 1 > 13 hd268dg.prod.mediav.com:38740 25 s 1 0 1 340.0 > MB / 315 5.7 MB / 1 > 14 hd258dg.prod.mediav.com:34498 22 s 1 0 1 339.8 > MB / 315 5.7 MB / 1 > 16 hd268dg.prod.mediav.com:27714 25 s 1 0 1 341.1 > MB / 316 5.7 MB / 1 > 17 hd258dg.prod.mediav.com:36584 22 s 1 0 1 340.7 > MB / 315 5.7 MB / 1 > 18 hd278dg.prod.mediav.com:37627 16 s 1 0 1 340.8 > MB / 315 5.7 MB / 1 > 19 hd268dg.prod.mediav.com:5938 25 s 1 0 1 339.8 > MB / 315 5.7 MB / 1 > 2 hd258dg.prod.mediav.com:46592 2.3 min 1 0 1 > 190.8 MB / 628778 6.5 MB / 1 > 20 hd258dg.prod.mediav.com:29043 2.2 min 1 0 1 > 190.5 MB / 627972 6.5 MB / 1 > You can see the records in executor 2 and executor 20 is too much with > respect to the storage. I think some bugs must exists during caching stage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org