[
https://issues.apache.org/jira/browse/PIG-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132216#comment-13132216
]
Alan Gates commented on PIG-2318:
---------------------------------
When running the following script (from e2e test LoadBinStorage_1):
{code}
register ./lib/java/testudf.jar;a = load
'/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
b = foreach a generate name, org.apache.pig.test.udf.evalfunc.Swap(name, age),
TOKENIZE((chararray)name),
org.apache.pig.test.udf.evalfunc.CreateMap((chararray)name, age);
store b into
'/user/pig/out/hortonal.1319153438/LoaderBinStorage_1.out.intermediate' using
BinStorage();
c = load
'/user/pig/out/hortonal.1319153438/LoaderBinStorage_1.out.intermediate' using
BinStorage();
store c into '/user/pig/out/hortonal.1319153438/LoaderBinStorage_1.out' using
org.apache.pig.test.udf.storefunc.StringStore();
{code}
the job fails to start, with the following exception:
{code}
java.lang.RuntimeException: Cannot instantiate:
org.apache.pig.test.udf.storefunc.StringStore
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:531)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:507)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070:
Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports:
[, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:516)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:528)
... 11 more
java.lang.RuntimeException: Cannot instantiate:
org.apache.pig.test.udf.storefunc.StringStore
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:531)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:507)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070:
Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports:
[, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:516)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:528)
... 11 more
java.lang.RuntimeException: Cannot instantiate:
org.apache.pig.test.udf.storefunc.StringStore
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:531)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:507)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070:
Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports:
[, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:516)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:528)
... 11 more
java.lang.RuntimeException: Cannot instantiate:
org.apache.pig.test.udf.storefunc.StringStore
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:531)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:507)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070:
Could not resolve org.apache.pig.test.udf.storefunc.StringStore using imports:
[, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:516)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:528)
... 11 more
{code}
> Push extra jars to distributed cache and use the classloader enxtension
> mechanism in PigContext to load them on the backend
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-2318
> URL: https://issues.apache.org/jira/browse/PIG-2318
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2318.patch, PIG-2318_2.patch
>
>
> This is related to PIG-2010 with a slightly different approach
> https://issues.apache.org/jira/browse/PIG-2010
> Currently Pig bundles up all dependencies in a single jar which is a lot of
> overhead when there are a lot of dependencies and short lived jobs. This
> patch instead pushes the dependencies to distributed cache and uses the
> PigContext classloading mechanism to make the UDFs available.
> Possible improvements: push jars to HDFS/distributed cache only once per
> script. have a cache on HDFS to avoid repeatedly pushing jars to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
