----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4651/#review8332 -----------------------------------------------------------
trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java <https://reviews.apache.org/r/4651/#comment17954> do we want to get rid of those lines? trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java <https://reviews.apache.org/r/4651/#comment17955> remove? trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java <https://reviews.apache.org/r/4651/#comment17956> you could also have a method isFixedSize() in TupleFactory. trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java <https://reviews.apache.org/r/4651/#comment17958> you could also do this at the same time you would have initialized the Schema based factory (line 144) if the assigment happens in the constructor you can even make inputTupleFactory final trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java <https://reviews.apache.org/r/4651/#comment17960> that's where -w does not do what you want :( trunk/src/org/apache/pig/data/BinInterSedes.java <https://reviews.apache.org/r/4651/#comment17962> where is this used? trunk/src/org/apache/pig/data/DataByteArray.java <https://reviews.apache.org/r/4651/#comment17965> http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Arrays.html#hashCode%28byte[]%29 I'm not sure I get why these should use different primes... trunk/src/org/apache/pig/data/TupleFactory.java <https://reviews.apache.org/r/4651/#comment17973> I prefer just Schema as a parameter here. I'll look again when you add SchemaTupleFrontend and Backend trunk/src/org/apache/pig/impl/PigContext.java <https://reviews.apache.org/r/4651/#comment17967> update javadoc trunk/src/org/apache/pig/impl/io/PigNullableWritable.java <https://reviews.apache.org/r/4651/#comment17969> those have setters, use them instead trunk/src/org/apache/pig/newplan/logical/expression/UserFuncExpression.java <https://reviews.apache.org/r/4651/#comment17971> move up the declaration of inputSchemaToGen and reuse it - Julien Le Dem On June 18, 2012, 5:22 p.m., Jonathan Coveney wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/4651/ > ----------------------------------------------------------- > > (Updated June 18, 2012, 5:22 p.m.) > > > Review request for pig and Julien Le Dem. > > > Description > ------- > > This work builds on Dmitriy's PrimitiveTuple work. The idea is that, knowing > the Schema on the frontend, we can code generate Tuples which can be used for > fun and profit. In rudimentary tests, the memory efficiency is 2-4x better, > and it's ~15% smaller serialized (heavily heavily depends on the data, > though). Need to do get/set tests, but assuming that it's on par (or even > faster) than Tuple, the memory gain is huge. > > Need to clean up the code and add tests. > > Right now, it generates a SchemaTuple for every inputSchema and outputSchema > given to UDF's. The next step is to make a SchemaBag, where I think the > serialization savings will be really huge. > > Needs tests and comments, but I want the code to settle a bit. > > > This addresses bug PIG-2632. > https://issues.apache.org/jira/browse/PIG-2632 > > > Diffs > ----- > > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 1351417 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java > 1351417 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java > 1351417 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java > 1351417 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java > 1351417 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java > 1351417 > trunk/src/org/apache/pig/data/BinInterSedes.java 1351417 > trunk/src/org/apache/pig/data/DataByteArray.java 1351417 > trunk/src/org/apache/pig/data/TupleFactory.java 1351417 > trunk/src/org/apache/pig/data/TypeAwareTuple.java 1351417 > trunk/src/org/apache/pig/impl/PigContext.java 1351417 > trunk/src/org/apache/pig/impl/io/NullableTuple.java 1351417 > trunk/src/org/apache/pig/impl/io/PigNullableWritable.java 1351417 > > trunk/src/org/apache/pig/newplan/logical/expression/ExpToPhyTranslationVisitor.java > 1351417 > trunk/src/org/apache/pig/newplan/logical/expression/UserFuncExpression.java > 1351417 > trunk/test/org/apache/pig/test/TestDataBag.java 1351417 > trunk/test/org/apache/pig/test/TestSchema.java 1351417 > > Diff: https://reviews.apache.org/r/4651/diff/ > > > Testing > ------- > > > Thanks, > > Jonathan Coveney > >
