Julien Le Dem created PIG-2699:
----------------------------------
Summary: Pig creates too many instances of Load and Store Funcs.
It should be 1 in the front-end and 1 in the backend
Key: PIG-2699
URL: https://issues.apache.org/jira/browse/PIG-2699
Project: Pig
Issue Type: Bug
Components: internal-udfs
Affects Versions: 0.10.0
Reporter: Julien Le Dem
Assignee: Julien Le Dem
Here is a patch to get it down to 3
Here is the report of the remaining calls:
{noformat}
3 instances of Loader
3 instances of Storer
all calls:
Loader(1).<init>
Loader(1).relativeToAbsolutePath
Storer(1).<init>
Storer(1).setStoreFuncUDFContextSignature
Storer(1).relToAbsPathForStoreLocation
Storer(1).setStoreFuncUDFContextSignature
Loader(1).getSchema
Loader(1).getSchema
Loader(1).getSchema
Loader(1).getSchema
Loader(1).getSchema
Loader(1).getSchema
Storer(1).setStoreFuncUDFContextSignature
Loader(1).getSchema
Loader(1).getSchema
Loader(1).getSchema
Storer(1).setStoreLocation
Storer(1).getOutputFormat
Loader(1).getSchema
Loader(1).getStatistics
Loader(1).setLocation
Storer(1).setStoreLocation
Storer(2).<init>
Storer(2).setStoreFuncUDFContextSignature
Storer(2).setStoreLocation
Storer(2).getOutputFormat
Loader(2).<init>
Loader(2).setUDFContextSignature
Loader(2).setLocation
Loader(2).getInputFormat
Storer(3).<init>
Storer(3).setStoreFuncUDFContextSignature
Storer(3).setStoreLocation
Storer(3).getOutputFormat
Storer(3).setStoreLocation
Loader(3).<init>
Loader(3).setUDFContextSignature
Loader(3).setLocation
Loader(3).getInputFormat
Storer(3).setStoreLocation
Storer(3).getOutputFormat
Storer(3).prepareToWrite
Loader(3).setUDFContextSignature
Loader(3).prepareToRead
Loader(3).getNext
Storer(3).putNext
Loader(3).getNext
Storer(3).putNext
Loader(3).getNext
Storer(3).putNext
Loader(3).getNext
Storer(3).setStoreLocation
Storer(3).setStoreLocation
Storer(1).setStoreLocation
constructor calls:
Loader(1).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:421)
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3170)
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1293)
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:791)
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:509)
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:384)
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:175)
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1602)
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1549)
org.apache.pig.PigServer.registerQuery(PigServer.java:534)
org.apache.pig.PigServer.registerQuery(PigServer.java:547)
Storer(1).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:477)
org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:6336)
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1337)
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:791)
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:509)
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:384)
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:175)
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1602)
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1549)
org.apache.pig.PigServer.registerQuery(PigServer.java:534)
org.apache.pig.PigServer.registerQuery(PigServer.java:547)
Storer(2).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:168)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecsHelper(PigOutputFormat.java:200)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:187)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:258)
Loader(2).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:254)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:258)
Storer(3).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:84)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:66)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
Loader(3).<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:565)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java:158)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:106)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
