Hello,
For test case TestPruneColumn:
public void testMapKeyInSplit1() throws Exception {
pigServer.registerQuery("A = load '"+
Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as
(m:map[]);");
pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
pigServer.registerQuery("D = join B by key1, C by key2;");
Iterator<Tuple> iter = pigServer.openIterator("D");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Map key required for A:
$0->[key2, key1]"}));
}
I have some questions:
1. What content will be stored in logFile in methond checkLogFileMessage?
2. With other JDK but not SUN JDK, I found the variable item contain string
like "Map key required for A: $0->[key1, key2]". I think this is caused by
pigServer.registerQuery method which invoke method parse. And parse method is
based on HashMap structure.Could you please tell me what HashMap methods will
be invoked and cause the different output LogicalPlan in parse method?
public LogicalPlan parse(String scope,
String query,
Map<LogicalOperator, LogicalPlan> aliases,
Map<OperatorKey, LogicalOperator> opTable,
Map<String, LogicalOperator> aliasOp,
int start,
Map<String, String> fileNameMap)
throws IOException, ParseException {
ByteArrayInputStream in = new ByteArrayInputStream(query.getBytes());
QueryParser parser = new QueryParser(in, pigContext, scope, aliases,
opTable,
aliasOp, start, fileNameMap);
return parser.Parse();
}
3.The same situation also happened in:
Testcase: testMapKey2 took 8.133 sec
Should be:"Columns pruned for A: $0",
"Map key required for A: $1->[key2, key1]"
But is:"Columns pruned for A: $0"
"Map key required for A: $1->[key1, key2]"
Testcase: testMapKey3 took 8.815 sec
Should be:{([key2#2,key1#1],1),([key2#4,key1#2],2)}
But is:{([key1#1,key2#2],1),([key1#2,key2#4],2)}
Testcase: testSharedSchemaObject took 9.15 sec
Should be:([2#1,1#1])
But is:([1#1,2#1])
Are these results also correct?
At 2011-08-19 01:50:20,"Thejas Nair" <[email protected]> wrote:
>yes, even in this case, both outputs look correct.
>-Thejas
>
>
>
>On 8/18/11 6:05 AM, lulynn_2008 wrote:
>> Thejas Nair,
>>
>> Thank you
>> Actually there are other unit tests failed because of different order
>> output. During parse the pig command, a HashMap variable "aliases" will
>> be used in
>> src-gen/org/apache/pig/impl/logicalLayer/parser/QueryParser.java. I
>> think this unit test failure is also caused by HashMap problem. What is
>> your opinion?
>> Here are two failure unit test examples:
>> TestLogToPhyCompiler: Query: split (load 'a') into x if $0 < '7', y if
>> $0 > '7';
>> Testcase: testSplit took 0.198 sec
>> FAILED
>> IBM JDK:
>> x: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Less Than[boolean] - Test-Plan-Builder-243
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-241
>> | |
>> | |---Constant(7) - Test-Plan-Builder-242
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>>
>> y: Filter[tuple] - Test-Plan-Builder-244
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-247
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-245
>> | |
>> | |---Constant(7) - Test-Plan-Builder-246
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>> SUN JDK:
>> y: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-243
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-241
>> | |
>> | |---Constant(7) - Test-Plan-Builder-242
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>>
>> x: Filter[tuple] - Test-Plan-Builder-244
>> | |
>> | Less Than[boolean] - Test-Plan-Builder-247
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-245
>> | |
>> | |---Constant(7) - Test-Plan-Builder-246
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>> TestMRCompiler(IBM JDK):
>> Testcase: testSortUDF1 took 0.02 sec
>> FAILED
>> null
>> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
>> - -18:
>> | ...> but
>> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
>> | ...>
>> junit.framework.ComparisonFailure: null
>> expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator])
>> - -18:
>> | ...> but
>> was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
>> | ...>
>> at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
>> at org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
>>
>>
>>
>>
>> At 2011-08-18 01:17:02,"Thejas Nair" <[email protected]> wrote:
>>>Since HashMap does not guarantee ordering, this test is not
>>>guaranteed
>>>to pass, ie the test is faulty and it needs to be fixed. The
>>>result from
>>>IBM JDK is also correct.
>>>An easy fix for the test case would be to test for both
>>>sequences of
>>>hash-map.
>>>Can you please open a jira ?
>>>
>>>-Thejas
>>>
>>>
>>>
>>>On 8/16/11 7:56 PM, lulynn_2008 wrote:
>>>> Hello,
>>>> I runTestDataModel test case with IBM JDK, and got the
>>>> following error:
>>>> Testcase: testTupleToString took 0.002 sec
>>>> FAILED
>>>> toString expected:<...ad a little
>>>> lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but
>>>> was:<...ad a little
>>>> lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>>> junit.framework.ComparisonFailure: toString expected:<...ad a little
>>>> lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but
>>>> was:<...ad a little
>>>> lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
>>>> at
>>>> org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
>>>> This is because IBM HashMap.toString() output different with SUN
>>>> HashMap.toString(). My question is:
>>>> Should the output must be the one expected?
>>>> Are the output expected based on SUN JDK?
>>>> Does the failed test case mean pig-0.8.1 can not work with IBM
>>>> JDk?
>>>> Thank you.
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
