Re: Reporting serialized task size after task broadcast change?
I thought we could see this on the Spark Web UI storage tab. May be I was looking at something else too. On Sep 11, 2014, at 8:47 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hmm, well I can't find it now, must have been hallucinating. Do you know off the top of your head where I'd be able to find the size to log it? On Thu, Sep 11, 2014 at 6:33 PM, Reynold Xin r...@databricks.com wrote: I didn't know about that On Thu, Sep 11, 2014 at 6:29 PM, Sandy Ryza sandy.r...@cloudera.com wrote: It used to be available on the UI, no? On Thu, Sep 11, 2014 at 6:26 PM, Reynold Xin r...@databricks.com wrote: I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Reporting serialized task size after task broadcast change?
After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
Re: Reporting serialized task size after task broadcast change?
I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
Re: Reporting serialized task size after task broadcast change?
It used to be available on the UI, no? On Thu, Sep 11, 2014 at 6:26 PM, Reynold Xin r...@databricks.com wrote: I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
Re: Reporting serialized task size after task broadcast change?
I didn't know about that On Thu, Sep 11, 2014 at 6:29 PM, Sandy Ryza sandy.r...@cloudera.com wrote: It used to be available on the UI, no? On Thu, Sep 11, 2014 at 6:26 PM, Reynold Xin r...@databricks.com wrote: I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
Re: Reporting serialized task size after task broadcast change?
Hmm, well I can't find it now, must have been hallucinating. Do you know off the top of your head where I'd be able to find the size to log it? On Thu, Sep 11, 2014 at 6:33 PM, Reynold Xin r...@databricks.com wrote: I didn't know about that On Thu, Sep 11, 2014 at 6:29 PM, Sandy Ryza sandy.r...@cloudera.com wrote: It used to be available on the UI, no? On Thu, Sep 11, 2014 at 6:26 PM, Reynold Xin r...@databricks.com wrote: I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
Re: Reporting serialized task size after task broadcast change?
It is in the dag scheduler. Look for broadcast. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com wrote: Hmm, well I can't find it now, must have been hallucinating. Do you know off the top of your head where I'd be able to find the size to log it? On Thu, Sep 11, 2014 at 6:33 PM, Reynold Xin r...@databricks.com javascript:_e(%7B%7D,'cvml','r...@databricks.com'); wrote: I didn't know about that On Thu, Sep 11, 2014 at 6:29 PM, Sandy Ryza sandy.r...@cloudera.com javascript:_e(%7B%7D,'cvml','sandy.r...@cloudera.com'); wrote: It used to be available on the UI, no? On Thu, Sep 11, 2014 at 6:26 PM, Reynold Xin r...@databricks.com javascript:_e(%7B%7D,'cvml','r...@databricks.com'); wrote: I don't think so. We should probably add a line to log it. On Thursday, September 11, 2014, Sandy Ryza sandy.r...@cloudera.com javascript:_e(%7B%7D,'cvml','sandy.r...@cloudera.com'); wrote: After the change to broadcast all task data, is there any easy way to discover the serialized size of the data getting sent down for a task? thanks, -Sandy
