And another:
https://stackoverflow.com/questions/55748746/issues-with-dynamic-destinations-in-dataflow
On Thu, Nov 14, 2019 at 1:35 AM Kenneth Knowles wrote:
>
>
> On Fri, Nov 8, 2019 at 9:44 AM Steve Niemitz wrote:
>
>> Yeah that looks like what I had in mind too. I think the most useful
>> n
On Fri, Nov 8, 2019 at 9:44 AM Steve Niemitz wrote:
> Yeah that looks like what I had in mind too. I think the most useful
> notification output would be a KV of (K, summary)?
>
Sounds about right. Some use cases may not care about the summary, but just
the notification. But for most runners pa
Yeah that looks like what I had in mind too. I think the most useful
notification output would be a KV of (K, summary)?
On Fri, Nov 8, 2019 at 12:38 PM Kenneth Knowles wrote:
> This sounds like a useful feature, if I understand it: a generic transform
> (build on a generic stateful DoFn) where
This sounds like a useful feature, if I understand it: a generic transform
(build on a generic stateful DoFn) where the end-user provides a monotonic
predicate over the input it has seen. It emits a notification exactly once
when the predicate is first satisfied. To be efficient, it will also need
Interestingly enough, we just had a use case come up that I think could
have been solved by finishing triggers.
Basically, we want to emit a notification when a certain threshold is
reached (in this case, we saw at least N elements for a given key), and
then never notify again within that window.
By the way, adding this guard uncovered two bugs in Beam's Java codebase,
luckily only benchmarks and tests. There were *no* non-buggy instances of a
finishing trigger. They both declare allowed lateness that is never used.
Nexmark query 10:
// Clear fancy triggering from above.
.
Opened https://github.com/apache/beam/pull/9960 for this idea. This will
alert users to broken pipelines and force them to alter them.
Kenn
On Thu, Oct 31, 2019 at 2:12 PM Kenneth Knowles wrote:
> On Thu, Oct 31, 2019 at 2:11 AM Jan Lukavský wrote:
>
>> Hi Kenn,
>>
>> does there still remain s
On Thu, Oct 31, 2019 at 2:11 AM Jan Lukavský wrote:
> Hi Kenn,
>
> does there still remain some use for trigger to finish? If we don't drop
> data, would it still be of any use to users? If not, would it be better
> to just remove the functionality completely, so that users who use it
> (and it w
Hi Kenn,
does there still remain some use for trigger to finish? If we don't drop
data, would it still be of any use to users? If not, would it be better
to just remove the functionality completely, so that users who use it
(and it will possibly break for them) are aware of it at compile time?
