[
https://issues.apache.org/jira/browse/SPARK-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362867#comment-14362867
]
Saisai Shao edited comment on SPARK-5523 at 3/16/15 7:36 AM:
-------------------------------------------------------------
After investigating a little on the implementation details of host in
{{TaskMetrics}} and {{TaskInfo}}.
I think for {{TaskInfo}}, Spark already uses {{hostPortParseResults}} in
Utils.scala to cache the host name, so we don't need to do this again. Also
{{TaskInfo}} will only reside in driver, so RPC related object recreation is
not existed.
For {{TaskMetrics}}, the problem lies in deserialization, mainly we could
rewrite {{readObject()}} function to use a cached host name like this way:
{code}
@throws(classOf[IOException])
private def readObject(in: ObjectInputStream): Unit = Utils.tryOrIOException {
in.defaultReadObject()
_hostname = getHostFromPool(_hostname)
}
{code}
The question is that we still create a {{_hostname}} String object, though very
short-life. I'm not sure is there a way we could reuse the cached hostname
without even creating a very short-life one. Any suggestions? Thanks a lot.
was (Author: jerryshao):
After investigating a little on the implementation details of host in
{{TaskMetrics}} and {{TaskInfo}}.
I think for {{TaskInfo}}, Spark already uses {{hostPortParseResults}} in
Utils.scala to cache the host name, so we don't need to do this again. Also
{{TaskInfo}} will only reside in driver, so RPC related object recreation is
not existed.
For {{TaskMetrics}}, the problem lies in deserialization, mainly we could
rewrite {{readObject()}} function to use a cached host name like this way:
{code}
@throws(classOf[IOException])
private def readObject(in: ObjectInputStream): Unit = Utils.tryOrIOException {
in.defaultReadObject()
_hostname = getHostFromPool(_hostname)
}
{code}
The question is that we still create a {{_hostname}} String object, though very
shot life. I'm not sure is there a way we could reuse the cached hostname
without even creating a very short-life one. Any suggestions? Thanks a lot.
> TaskMetrics and TaskInfo have innumerable copies of the hostname string
> -----------------------------------------------------------------------
>
> Key: SPARK-5523
> URL: https://issues.apache.org/jira/browse/SPARK-5523
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Streaming
> Reporter: Tathagata Das
>
> TaskMetrics and TaskInfo objects have the hostname associated with the task.
> As these are created (directly or through deserialization of RPC messages),
> each of them have a separate String object for the hostname even though most
> of them have the same string data in them. This results in thousands of
> string objects, increasing memory requirement of the driver.
> This can be easily deduped when deserializing a TaskMetrics object, or when
> creating a TaskInfo object.
> This affects streaming particularly bad due to the rate of job/stage/task
> generation.
> For solution, see how this dedup is done for StorageLevel.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/StorageLevel.scala#L226
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
