Hi Ana,
in general, worker nodes do not share any state, and cannot themselves
decide which work to accept and which to reject. How the work is
distributed to downstream processing is defined by a runner, not the
Beam model. On the other hand, what you ask for might be possibly
accomplished using a grouping operation - either a GroupByKey or a
stateful DoFn might help you with that. Can you further describe your
intent?
Best,
Jan
On 9/7/21 12:32 PM, Ana Markovic wrote:
To whom this may concern,
I've been looking into polyglot data processing frameworks recently,
and I read Beam's documentation as well as developed a few examples to
get some hands-on experience. I've been wondering, and I haven't found
this in the documentation, is there a way to set up worker nodes so
they are "opinionated" or "smart" in a sense that they can decide for
themselves which jobs they will perform? For example, in a word count
example, an opinionated worker node could only decide to monitor
occurrences of a specific word if it's among the node's favourite words.
I hope I explained it well, but please let me know if more details are
needed to answer this question.
Thankful in advance,
Ana
