[
https://issues.apache.org/jira/browse/PIG-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310108#comment-15310108
]
Pallavi Rao commented on PIG-4893:
----------------------------------
+1 for addressing this.
When I had noticed this problem, one way I thought we could solve the problem
was by:
1. Excluding certain jars we know for certain are not needed.
2. Also, provide an option to user to specify an environment variable which
contains the list of jars that needs to be loaded to dcache. We should default
to our list, if this env. variable is not specified.
> Task deserialization time is too long for spark on yarn mode
> ------------------------------------------------------------
>
> Key: PIG-4893
> URL: https://issues.apache.org/jira/browse/PIG-4893
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: time.PNG
>
>
> I found the task deserialization time is a bit long when i run any scripts of
> pigmix in spark on yarn mode. see the attachment picture. The duration time
> is 3s but the task deserialization is 13s.
> My env is hadoop2.6+spark1.6.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
