Re: Can an Aggregate the key from a WindowedStream.aggregate()
Hi Stephen, Chesney was right, you will have to use a more complex version of the window processing function. Perhaps your goal can be achieve by this specific function with incremental aggregation [1]. If not you can always use the regular process window function [2]. Both of these methods have access to the KEY information you required. Thanks, Rong [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation [2] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#processwindowfunction On Sun, Feb 10, 2019 at 11:29 AM Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > > > On Sun, 10 Feb 2019 at 09:48, Chesnay Schepler wrote: > >> There are also versions of WindowedStream#aggregate that accept an >> additional WindowFunction/ProcessWindowFunction, which do have access to >> the key via apply()/process() respectively. These functions are called >> post aggregation. >> > > Cool I'll chase those down > > >> >> On 08.02.2019 18:24, stephen.alan.conno...@gmail.com wrote: >> > If I write my aggregation logic as a WindowFunction then I get access >> to the key as the first parameter in WindowFunction.apply(...) however the >> Javadocs for calling WindowedStream.apply(WindowFunction) state: >> > >> >> Note that this function requires that all data in the windows is >> buffered until the window >> >> is evaluated, as the function provides no means of incremental >> aggregation. >> > Which sounds bad. >> > >> > It seems the recommended alternative is to use one of the >> WindowFunction.aggregate(AggregateFunction) however I cannot see how to get >> access to the key... >> > >> > Is my only solution to transform my data into a Tuple if I need access >> to the key post aggregation? >> > >> > Thanks in advance >> > >> > -stephenc >> > >> >>
Re: Can an Aggregate the key from a WindowedStream.aggregate()
On Sun, 10 Feb 2019 at 09:48, Chesnay Schepler wrote: > There are also versions of WindowedStream#aggregate that accept an > additional WindowFunction/ProcessWindowFunction, which do have access to > the key via apply()/process() respectively. These functions are called > post aggregation. > Cool I'll chase those down > > On 08.02.2019 18:24, stephen.alan.conno...@gmail.com wrote: > > If I write my aggregation logic as a WindowFunction then I get access to > the key as the first parameter in WindowFunction.apply(...) however the > Javadocs for calling WindowedStream.apply(WindowFunction) state: > > > >> Note that this function requires that all data in the windows is > buffered until the window > >> is evaluated, as the function provides no means of incremental > aggregation. > > Which sounds bad. > > > > It seems the recommended alternative is to use one of the > WindowFunction.aggregate(AggregateFunction) however I cannot see how to get > access to the key... > > > > Is my only solution to transform my data into a Tuple if I need access > to the key post aggregation? > > > > Thanks in advance > > > > -stephenc > > > >
Re: Can an Aggregate the key from a WindowedStream.aggregate()
There are also versions of WindowedStream#aggregate that accept an additional WindowFunction/ProcessWindowFunction, which do have access to the key via apply()/process() respectively. These functions are called post aggregation. On 08.02.2019 18:24, stephen.alan.conno...@gmail.com wrote: If I write my aggregation logic as a WindowFunction then I get access to the key as the first parameter in WindowFunction.apply(...) however the Javadocs for calling WindowedStream.apply(WindowFunction) state: Note that this function requires that all data in the windows is buffered until the window is evaluated, as the function provides no means of incremental aggregation. Which sounds bad. It seems the recommended alternative is to use one of the WindowFunction.aggregate(AggregateFunction) however I cannot see how to get access to the key... Is my only solution to transform my data into a Tuple if I need access to the key post aggregation? Thanks in advance -stephenc
Can an Aggregate the key from a WindowedStream.aggregate()
If I write my aggregation logic as a WindowFunction then I get access to the key as the first parameter in WindowFunction.apply(...) however the Javadocs for calling WindowedStream.apply(WindowFunction) state: > Note that this function requires that all data in the windows is buffered > until the window > is evaluated, as the function provides no means of incremental aggregation. Which sounds bad. It seems the recommended alternative is to use one of the WindowFunction.aggregate(AggregateFunction) however I cannot see how to get access to the key... Is my only solution to transform my data into a Tuple if I need access to the key post aggregation? Thanks in advance -stephenc
