And both Wenhai and Preston have examples of doing the
fan-in-and-compute/fan-back-out pattern with blocking until the latter
part is done - Wenhai for finding range split points for parallel
sorting and Preston for similar things that arise in interval joins.
Can you guys chime in when you have a chance? (Preston may be busy from
what I saw on Skype on Friday :-), with congrats being due!)
On 10/11/16 12:22 AM, Jianfeng Jia wrote:
Based on the described example, it seems possible to implement it in one job by
You can force that merge-BF-operator only runs in one partition by using
On Oct 10, 2016, at 11:43 PM, mingda li <limingda1...@gmail.com> wrote:
Yeah, that will be easier. But for example, we have N nodes and in each
node, it will generate a Bloom Filter(BF) for its own data. We need to send
these BFs to one node for constructing a complete BF and then send the BF
back to each node. I am not sure we can use multiple stage job for this,
because there should be a 1->N and a N->1 connecter among nodes. If in one
job, there may be no way to transfer data among nodes.
This is my idea. If this can be implemented by one multiple stage job, that
will decrease a lot of my work :-)
On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtab...@gmail.com> wrote:
Is there a reason for wanting two jobs? I would think that one multiple
stage job would be preferable.
On Oct 10, 2016 1:21 PM, "mingda li" <limingda1...@gmail.com> wrote:
Oh, thanks Kim~
On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangs...@gmail.com> wrote:
Forwarded to dev.
---------- Forwarded message ----------
From: mingda li <limingda1...@gmail.com>
Date: Mon, Oct 10, 2016 at 11:21 AM
Subject: Let one Operator finished the job before another one begin in
Now,I am trying to build a Bloom Filter(BF) before join. The BF is
each node and sent to one node to combine. I want to set a stop sign
before sending the BF in each node. The stop sign means it can only
the BF after it is build.
The class HyracksConnection.waitForCompletion may help this. But I am
sure how to use it.
Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
Has anyone ever used the HyracksConnection.waitForCompletion?