Hi,

I used the combineGroup function to reduce groups of a very large dataset.
By modifying the parallelism to 1 I have a different results with a
parallelism to 8, Knowing that the good results are those obtained with the
parallelism with 1.

I also used table api to group dataset and select sum of column, i also had
the same result with parallelism to 8 and combineGroup.

My combineGroup function :

public class SumGroup extends Resampling implements
GroupReduceFunction<Row, Row>{

      @Override
       public void reduce(Iterable<Row> values, Collector<Row> out) throws
Exception {
             Iterator<Row> itr = values.iterator();
             double sum = 0;
             Row row = null;
             while(itr.hasNext()) {
                         row = itr.next();
                         sum = sum + (double) row.getField(indexField);
             }
             row.setField(indexField, sum);
            out.collect(row);
      }
   }
I call this function like this :
DataSet<Row> ds = env.toDataSet(inputTable(env), Row.class)
                      .groupBy(groupFields.stream().toArray(String[] ::
new))
                             .sortGroup(sortField, Order.ASCENDING)
                                    .reduceGroup(new
SumGroup()).returns(getOutputType());

results with parallelism to 1 :
24000 cluster 1 28/02/2017 06:00:00
21000 cluster 1 31/03/2017 06:00:00
15000 cluster 1 30/04/2017 06:00:00
10000 cluster 1 31/05/2017 06:00:00

results with parallelism to 8 :
22000 cluster 1 28/02/2017 06:00:00
4350 cluster 1 31/03/2017 06:00:00
14000 cluster 1 30/04/2017 06:00:00
2256 cluster 1 31/05/2017 06:00:00
I do not know how to make no difference with the modification of
parallelism and have same results with parallelism to 1.

can someone have any idea to help me please ?

thank you in advance !!

Anissa

Reply via email to