Hi Max,
can I create a Jira and work on this ?
Thanks,
Regards
JB
On 10/18/2016 03:04 PM, Maximilian Michels wrote:
+1 for removing the ones which are merely duplicates.
-Max
On Mon, Oct 17, 2016 at 3:11 PM, Jean-Baptiste Onofré <[email protected]> wrote:
Fully agree.
It's what we discussed in the "refactoring Flink runner" PR ;)
Please, let me know if I can help there.
Regards
JB
On 10/17/2016 03:10 PM, Aljoscha Krettek wrote:
IMHO, now might be a good idea to remove the custom Flink examples that
started out as copies of the examples of the Dataflow SDK.
On Mon, 17 Oct 2016 at 14:46 Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Hi Trevor,
The examples used to be working but they are not anymore due to SDK
changes. Unfortunately, they are not automatically tested. We should
change that.
You can make the examples work if you use a bounded source, e.g. set
`.withMaxNumRecords(100)` on the socket source.
Cheers,
Max
On Sun, Oct 16, 2016 at 10:43 PM, Trevor Grant
<[email protected] <mailto:[email protected]>> wrote:
> So to be clear-
>
> The following currently merged examples aren't supposed to be
runnable
>
https://github.com/apache/incubator-beam/blob/f62d04e22679ee2ac19e3ae37dec487d953d51c1/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java
>
https://github.com/apache/incubator-beam/blob/f62d04e22679ee2ac19e3ae37dec487d953d51c1/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java
>
> Because those throw the same error re: Write / Bounded collections.
>
> tg
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> "Fortunate is he, who is able to know the causes of things."
-Virgil
>
>
> On Fri, Oct 14, 2016 at 8:09 PM, Kenneth Knowles <[email protected]
<mailto:[email protected]>> wrote:
>>
>> Hi Trevor,
>>
>> The problem is that "Write" is an old name that should be changed
to
>> "BoundedWrite" (actually it is even more specific). In fact, it
re-windows
>> into the global window and removes all triggering, so it is
suitable only
>> for bounded PCollections where this will ensure all the data
arrives at
>> once. Hence the error message.
>>
>> For a streaming write, I'd recommend just performing the write in a
>> ParDo(DoFn) with probably some prep to get the data ready for
idempotent
>> writing. If you look around the codebase you'll see some examples
of this.
>> Also note that Write isn't a primitive but just a certain pattern
of how you
>> can use ParDo and side inputs.
>>
>> Kenn
>>
>>
>> On Fri, Oct 14, 2016, 17:16 Trevor Grant
<[email protected] <mailto:[email protected]>> wrote:
>>>
>>> Hi,
>>>
>>> Trying to write a Twitter exampe on the Flink streamer-
>>>
>>>
>>>
https://github.com/rawkintrevo/incubator-beam/blob/flink-twitter-example/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/TwitterWindowedWordCountExamples.java
>>>
>>> I'm getting an error
>>> Caused by: java.lang.IllegalArgumentException: Write can only be
applied
>>> to a Bounded PCollection
>>>
>>> I was nearly copy and pasting from the other word count example,
every
>>> thing looks good in intellij. Can't figure out for the life of
me what I'm
>>> doing wrong here.
>>>
>>> I am running the job against a flink cluster (uploaded via
web-UI) if
>>> that is informative...
>>>
>>> Thanks, I plan on adding the example back in once it is done
(still have
>>> to parse the tweets).
>>>
>>> tg
>>>
>>>
>>> Trevor Grant
>>> Data Scientist
>>> https://github.com/rawkintrevo
>>> http://stackexchange.com/users/3002022/rawkintrevo
>>> http://trevorgrant.org
>>>
>>> "Fortunate is he, who is able to know the causes of things."
-Virgil
>>>
>
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
