And we could also combine several reduceFields(), similar to how we combine aggregates, for example andReduceField().
On Sat, Nov 1, 2014 at 8:50 PM, Stephan Ewen <[email protected]> wrote: > Hi Manu! > > I think the change to explicitly expose the group reduce key is separate > from the overhaul of the aggregation functions. > > I have added a comment to the issue, outlining the ideas gathered to far. > > There is also an interesting follow-up possible: Reduce on individual > fields. Frequently, the reduce affects only one field, as for example in > Spark's reduceByKey, which is a special case for key-value pairs. We could > add a nice generic version "reduceField(fieldExpression, ReduceFunction<f>, > )". > > An example for how this may look in a program is: > > DataSet<MyType> pairs = ... > pairs.groupBy("name") > . reduceField("counts", (a, b) -> a + b); > > > Stephan > > > > > On Sat, Nov 1, 2014 at 12:01 PM, Manu Kaul <[email protected]> wrote: > >> Hi All, >> I shall be joining DIMA as a Senior Researcher from >> Tuesday 4 Nov onwards. >> >> I have more of a background in C++ (for OO) and languages like >> SML and Haskell (for FP). So I am learning the Scala syntax at >> the moment. >> >> I came across the following improvement request: >> https://issues.apache.org/jira/browse/FLINK-553 >> >> Seeing that I have not yet had a chance to look at the Scala API >> and internals of Flink, would it be ok for me to work on this improvement >> or is this part of a bigger change? I see that most of the aggregation >> functions will anyway need a major overhaul, so I am not sure if this >> change also falls under the same ambit? >> >> I think the idea might be to return more data from the first function so >> that the next function in line in the composition has more to take as >> input. I am not sure if there are many users already using Flink >> because such a change would obviously break a lot of code. >> >> If this issue has other dependencies that I am not seeing at the >> moment, then please feel free to suggest some other Scala related >> change. >> >> Thanks, >> Manu >> >> >> -- >> >> The greater danger for most of us lies not in setting our aim too high and >> falling short; but in setting our aim too low, and achieving our mark. >> - Michelangelo >>
