thanks Ak, thanks for your idea. I had tried using spark to do what the
shell did. However it is not fast enough as I expected and not very easy.
Thanksamp;Best regards!
San.Luo
- 原始邮件 -
发件人：Akhil Das ak...@sigmoidanalytics.com
收件人：罗辉
Hi 罗辉
I think you interpret the logs wrong.
Your program actually runs from this point: (Rest of them are just starting
up stuffs and connecting)
15/06/08 16:14:22 INFO broadcast.TorrentBroadcast: Started reading
broadcast variable 0
15/06/08 16:14:23 INFO storage.MemoryStore:
hi akhil
Not exactly ,the task took 54s to finish, started from 16:14:02 and ended at
16:14:56.
within this 54s , it needs 19s to store value in memory, which started from
16:14:23 and ended at 16:14:42. I think this is the most time-wasting part of
this task ,also unreasonable.You may check
Only 1 minor GC, 0.07s.
Thanksamp;Best regards!
San.Luo
- 原始邮件 -
发件人：Akhil Das ak...@sigmoidanalytics.com
收件人：罗辉 luohui20...@sina.com
抄送人：user user@spark.apache.org
主题：Re: How to decrease the time of storing block in memory
日期：2015年06月09日 15点02分
Is it that task taking 19s? It won't be simply taking 19s to store 2KB of
data into memory there could be other operations happening too (the
transformations that you are doing), It would be good if you can paste the
code snippet that you are running to have a better understanding.
Thanks
Best
May be you should check in your driver UI and see if there's any GC time
involved etc.
Thanks
Best Regards
On Mon, Jun 8, 2015 at 5:45 PM, luohui20...@sina.com wrote:
hi there
I am trying to descrease my app's running time in worker node. I
checked the log and found the most
hi there I am trying to descrease my app's running time in worker node. I
checked the log and found the most time-wasting part is below:15/06/08 16:14:23
INFO storage.MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 2.1 KB, free 353.3 MB)
15/06/08 16:14:42 INFO
