[ 
https://issues.apache.org/jira/browse/PIG-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729882#action_12729882
 ] 

Pradeep Kamath commented on PIG-880:
------------------------------------

The root cause of this issue is that in interpreting map data, PigStorage 
returns values in the map to be of the type that it deduces based on the data. 
So string data for values are returned as String, integer values are returned 
as Integer. However the logical layer in Pig assumes the type of the values in 
the map to be ByteArray since it cannot assume any type. If one of the sampled 
values forming the quantile list is a null, it is assumed to be of type of the 
reduce key of the final order by job. In this case, since the order by key is 
smap#'name', it is thought to be of type ByteArray. However the values 
resulting from the map lookup are actually of type String.  This mismatch 
results in the above exception - if nulls are filtered out, map.collect() fails 
because hadoop thinks the map key type is bytearray but it gets a Text (string).

A proposal to fix this is to Change TextDataParser which is used by PigStorage 
for reading map data to return ByteArray type for the values in the map.

Thoughts?



> Order by is borken with complex fields
> --------------------------------------
>
>                 Key: PIG-880
>                 URL: https://issues.apache.org/jira/browse/PIG-880
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>            Reporter: Olga Natkovich
>             Fix For: 0.4.0
>
>
> Pig script:
> a = load 'studentcomplextab10k' as (smap:map[],c2,c3);
> f = foreach a generate smap#'name, smap#'age', smap#'gpa' ;            
> s = order f by $0;           
> store s into 'sc.out'         
> Stack:
> Caused by: java.lang.ArrayStoreException
>         at java.lang.System.arraycopy(Native Method)
>         at java.util.Arrays.copyOf(Arrays.java:2763)
>         at java.util.ArrayList.toArray(ArrayList.java:305)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.convertToArray(WeightedRangePartitioner.java:154)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.configure(WeightedRangePartitioner.java:96)
>         ... 5 more
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:230)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:179)
>         at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:204)
>         at 
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:265)
>         at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:769)
>         at org.apache.pig.PigServer.execute(PigServer.java:762)
>         at org.apache.pig.PigServer.access$100(PigServer.java:91)
>         at org.apache.pig.PigServer$Graph.execute(PigServer.java:933)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:245)
>         at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:140)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:88)
>         at org.apache.pig.Main.main(Main.java:389)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to