[
https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shravan Matthur Narayanamurthy updated PIG-162:
-----------------------------------------------
Attachment: incr7.patch
Patch with the following changes:
MR version of Sort supported (has ugly hand built physical plans for the
quantile job & sort job)
Made changes to the PlanPrinter to also visit and print the nested expr plans
Made changes to the name() methods of some Physical Operators to display more
information.
Changed POProject to include a StarSpec. Included a boolean variable start to
denote if the POProject represents a star spec. If so in the getNext(Tuple)
method, passes the inp tuple as is instead of projecting something.
Changed POLocalRearrange by removing the Physical plan containing generate. It
now works on just a List of expr plans building key and value tuples on its
own. This curtails one's ability to supply flatten inside the local rearrange
but I am really not sure if/how it is supported currently and whether it really
makes sense.
Made changes to MRCompiler to support POSort, to compute the requested
parallelism as the max of req par of all the operators inside a Map-Reduce job.
Also included are changes to MRCompiler which create the set of UDFs that are
associated with a job so that they can be loaded into the job jar file. To
achieve this, have introduced two visitor classes UDFFinder & UDFFinderForExp
Converted POUserFunc and POUserComparisonFunc into ExpressionOps.
Changed the launcher thingy to have non static protected launch method. Also
made the launcher's implicit constructor protected so that direct launcher
objects can't be created.
Have testcases to check changes to MRCompiler for Sort, requested parallelism &
UDF computation.
Include shubham's test cases for BinCond and Negative in the build.xml which he
probably forgot to do.
> Rework mapreduce submission and monitoring
> ------------------------------------------
>
> Key: PIG-162
> URL: https://issues.apache.org/jira/browse/PIG-162
> Project: Pig
> Issue Type: Sub-task
> Environment: This bug tracks works to rework the submission and
> monitoring interface to map reduce as described in
> http://wiki.apache.org/pig/PigTypesFunctionalSpec
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: incr7.patch, mapreduceJumbo.patch,
> mapreduceJumboWithComInc.patch, modJumbo.patch, split.png,
> TEST-org.apache.pig.test.TestMRCompiler.txt,
> TEST-org.apache.pig.test.TestMRCompiler.txt,
> TEST-org.apache.pig.test.TestUnion.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
