Dear Pig Expers,
I am quite new to Pig. I implemented some scripts and I am issuing
performance problems and it would be great to get some feedback or ideas on
how I could proceed to solve it.
I did implement a PIG script and when I execute the script it results in
many mappers for a specific step, but has only one reducer for that step
and so the Hadoop cluster is almost idle while this reducer is executed. In
order to better use the resources of the cluster I would like to have also
many reducers running in parallel.
Even if I set the parallelism in the Pip script using the SET
DEFAULT_PARALLEL command I still result in having only 1 reducer.
The code part issuing the problem is the following:
SET DEFAULT_PARALLEL 5;
inputData = LOAD 'input_data.txt' AS (group_name:chararray, item:int);
inputDataGrouped = GROUP inputData BY (group_name);
-- The GeneratePairsUDF generates a bag containing pairs of integers, e.g.
{(1, 5), (1, 8), ..., (8, 5)}
pairs = FOREACH inputDataGrouped GENERATE GeneratePairsUDF(inputData.item)
AS pairs_bag;
pairsFlat = FOREACH pairs GENERATE FLATTEN(pairs_bag) AS (item1:int,
item2:int);
The 'inputData' and 'inputDataGrouped' aliases are computed in the mapper.
The 'pairs' and 'pairsFlat' in the reducer.
If I change the script by removing the line with the FLATTEN command
(pairsFlat = FOREACH pairs GENERATE FLATTEN(pairs_bag) AS (item1:int,
item2:int);) then the execution results in 5 reducers
(and thus in a parallel execution):
It seems that the FLATTEN command is the problem and avoids that many
reducers are created.
Does anybody know how I could reach the same result of FLATTEN but having
the script being executed in parallel (with many reducers)?
Many thanks in advance for your support!
Best regards,
Christian
