And both Wenhai and Preston have examples of doing the fan-in-and-compute/fan-back-out pattern with blocking until the latter part is done - Wenhai for finding range split points for parallel sorting and Preston for similar things that arise in interval joins. Can you guys chime in when you have a chance? (Preston may be busy from what I saw on Skype on Friday :-), with congrats being due!)


On 10/11/16 12:22 AM, Jianfeng Jia wrote:
Based on the described example, it seems possible to implement it in one job by 
using MToNPartitioningConnectorDescriptor.
You can force that merge-BF-operator only runs in one partition by using 
PartitionConstraintHelper.addAbsoluteLocationConstraint() function.

On Oct 10, 2016, at 11:43 PM, mingda li <limingda1...@gmail.com> wrote:

Yeah, that will be easier. But for example, we have N nodes and in each
node, it will generate a Bloom Filter(BF) for its own data. We need to send
these BFs to one node for constructing a complete BF and then send the BF
back to each node. I am not sure we can use multiple stage job for this,
because there should be a 1->N and a N->1 connecter among nodes. If in one
job, there may be no way to transfer data among nodes.
This is my idea. If this can be implemented by one multiple stage job, that
will decrease a lot of my work :-)

Bests,
Mingda

On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtab...@gmail.com> wrote:

Is there a reason for wanting two jobs?  I would think that one multiple
stage job would be preferable.

On Oct 10, 2016 1:21 PM, "mingda li" <limingda1...@gmail.com> wrote:

Oh, thanks Kim~

On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangs...@gmail.com> wrote:

Forwarded to dev.

Best,
Taewoo

---------- Forwarded message ----------
From: mingda li <limingda1...@gmail.com>
Date: Mon, Oct 10, 2016 at 11:21 AM
Subject: Let one Operator finished the job before another one begin in
Hyracks
To: us...@asterixdb.apache.org


Hi,

Now,I am trying to build a Bloom Filter(BF) before join. The BF is
build
in
each node and sent to one node to combine. I want to set a stop sign
there
before sending the BF in each node. The stop sign means it can only
send
the BF after it is build.
The class HyracksConnection.waitForCompletion may help this. But I am
not
sure how to use it.
Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
jobidSendBF=hcc.startJob(); ?
Has anyone ever used the HyracksConnection.waitForCompletion?

Thanks,
Mingda


Reply via email to