Hi all:
I have 2 questiones about pig/test/e2e/pig/tests/streaming_local.conf:
1.
$cfg = {
'driver' => 'Pig',
'nummachines' => 5,
'groups' => [
{
# This group is for local mode testing
'name' => 'StreamingLocal',
'sortBenchmark' => 1,
'sortResults' => 1,
'floatpostprocess' => 1,
'delimiter' => ' ',
'tests' => [
{
#Section 1.1: perl script, no parameters
'num' => 1,
'execonly' => 'local', // this line
'pig' => q#
all e2e test cases are only executed in local mode now. Can these e2e tests run
in other mode, like "mapreduce","tez","spark"?
when i replace 'execonly'=>'local' with 'execonly'=>'spark', all cases pass
when POStream is implemented in spark mode.
I think we can remove 'execonly'=>'local' and can test these e2e tests in other
modes.
2. when using ship with streaming, it will send streaming binary and supporting
files, if any, from the client node to the compute nodes..I found we use "perl
./libexec/GroupBy.pl" in StreamingLocal_3.pig, this path is a relative path to
current executed path. can we use "perl GroupBy.pl" because i think the file
"./libexec/GroupBy.pl" has been shipped to compute nodes.
StreamingLocal_3.pig
/test/e2e/pigMD `perl ./libexec/GroupBy.pl '\t' 0` ship('./libexec/GroupBy.pl');
A = load './data/singlefile/studenttab10k';
B = group A by $0;
C = foreach B generate flatten(A);
D = stream C through CMD;
store D into
'./testout/root-1419582821-streaming_local.conf/StreamingLocal_3.out';
Best regards
Zhang,Liyun
