[
https://issues.apache.org/jira/browse/PIG-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136382#comment-13136382
]
Dmitriy V. Ryaboy commented on PIG-2318:
----------------------------------------
Looks like there's an edge case of a class in Pig needing a class found in the
pig jar, but not bundled into the job jar thanks to Pigs rebundling (and
packaging only prefixed classes). This affects some uses of HBaseStorage. We're
working on a fix (suspect just registering the pig jar itself will work. Or not
rebundling the pig jar in the first place? Why do we do that?)
> Push extra jars to distributed cache and use the classloader enxtension
> mechanism in PigContext to load them on the backend
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-2318
> URL: https://issues.apache.org/jira/browse/PIG-2318
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2318.patch, PIG-2318_2.patch, PIG-2318_3.patch,
> PIG-2318_4.patch
>
>
> This is related to PIG-2010 with a slightly different approach
> https://issues.apache.org/jira/browse/PIG-2010
> Currently Pig bundles up all dependencies in a single jar which is a lot of
> overhead when there are a lot of dependencies and short lived jobs. This
> patch instead pushes the dependencies to distributed cache and uses the
> PigContext classloading mechanism to make the UDFs available.
> Possible improvements: push jars to HDFS/distributed cache only once per
> script. have a cache on HDFS to avoid repeatedly pushing jars to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
