IMHO, now might be a good idea to remove the custom Flink examples that started out as copies of the examples of the Dataflow SDK.
On Mon, 17 Oct 2016 at 14:46 Maximilian Michels <[email protected]> wrote: > Hi Trevor, > > The examples used to be working but they are not anymore due to SDK > changes. Unfortunately, they are not automatically tested. We should > change that. > > You can make the examples work if you use a bounded source, e.g. set > `.withMaxNumRecords(100)` on the socket source. > > Cheers, > Max > > On Sun, Oct 16, 2016 at 10:43 PM, Trevor Grant <[email protected]> > wrote: > > So to be clear- > > > > The following currently merged examples aren't supposed to be runnable > > > https://github.com/apache/incubator-beam/blob/f62d04e22679ee2ac19e3ae37dec487d953d51c1/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java > > > https://github.com/apache/incubator-beam/blob/f62d04e22679ee2ac19e3ae37dec487d953d51c1/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java > > > > Because those throw the same error re: Write / Bounded collections. > > > > tg > > > > Trevor Grant > > Data Scientist > > https://github.com/rawkintrevo > > http://stackexchange.com/users/3002022/rawkintrevo > > http://trevorgrant.org > > > > "Fortunate is he, who is able to know the causes of things." -Virgil > > > > > > On Fri, Oct 14, 2016 at 8:09 PM, Kenneth Knowles <[email protected]> wrote: > >> > >> Hi Trevor, > >> > >> The problem is that "Write" is an old name that should be changed to > >> "BoundedWrite" (actually it is even more specific). In fact, it > re-windows > >> into the global window and removes all triggering, so it is suitable > only > >> for bounded PCollections where this will ensure all the data arrives at > >> once. Hence the error message. > >> > >> For a streaming write, I'd recommend just performing the write in a > >> ParDo(DoFn) with probably some prep to get the data ready for idempotent > >> writing. If you look around the codebase you'll see some examples of > this. > >> Also note that Write isn't a primitive but just a certain pattern of > how you > >> can use ParDo and side inputs. > >> > >> Kenn > >> > >> > >> On Fri, Oct 14, 2016, 17:16 Trevor Grant <[email protected]> > wrote: > >>> > >>> Hi, > >>> > >>> Trying to write a Twitter exampe on the Flink streamer- > >>> > >>> > >>> > https://github.com/rawkintrevo/incubator-beam/blob/flink-twitter-example/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/TwitterWindowedWordCountExamples.java > >>> > >>> I'm getting an error > >>> Caused by: java.lang.IllegalArgumentException: Write can only be > applied > >>> to a Bounded PCollection > >>> > >>> I was nearly copy and pasting from the other word count example, every > >>> thing looks good in intellij. Can't figure out for the life of me what > I'm > >>> doing wrong here. > >>> > >>> I am running the job against a flink cluster (uploaded via web-UI) if > >>> that is informative... > >>> > >>> Thanks, I plan on adding the example back in once it is done (still > have > >>> to parse the tweets). > >>> > >>> tg > >>> > >>> > >>> Trevor Grant > >>> Data Scientist > >>> https://github.com/rawkintrevo > >>> http://stackexchange.com/users/3002022/rawkintrevo > >>> http://trevorgrant.org > >>> > >>> "Fortunate is he, who is able to know the causes of things." -Virgil > >>> > > >
