And both Wenhai and Preston have examples of doing the
fan-in-and-compute/fan-back-out pattern with blocking until the latter
part is done - Wenhai for finding range split points for parallel
sorting and Preston for similar things that arise in interval joins.
Can you guys chime in when you have a chance? (Preston may be busy from
what I saw on Skype on Friday :-), with congrats being due!)
On 10/11/16 12:22 AM, Jianfeng Jia wrote:
Based on the described example, it seems possible to implement it in one job by
using MToNPartitioningConnectorDescriptor.
You can force that merge-BF-operator only runs in one partition by using
PartitionConstraintHelper.addAbsoluteLocationConstraint() function.
On Oct 10, 2016, at 11:43 PM, mingda li <[email protected]> wrote:
Yeah, that will be easier. But for example, we have N nodes and in each
node, it will generate a Bloom Filter(BF) for its own data. We need to send
these BFs to one node for constructing a complete BF and then send the BF
back to each node. I am not sure we can use multiple stage job for this,
because there should be a 1->N and a N->1 connecter among nodes. If in one
job, there may be no way to transfer data among nodes.
This is my idea. If this can be implemented by one multiple stage job, that
will decrease a lot of my work :-)
Bests,
Mingda
On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <[email protected]> wrote:
Is there a reason for wanting two jobs? I would think that one multiple
stage job would be preferable.
On Oct 10, 2016 1:21 PM, "mingda li" <[email protected]> wrote:
Oh, thanks Kim~
On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <[email protected]> wrote:
Forwarded to dev.
Best,
Taewoo
---------- Forwarded message ----------
From: mingda li <[email protected]>
Date: Mon, Oct 10, 2016 at 11:21 AM
Subject: Let one Operator finished the job before another one begin in
Hyracks
To: [email protected]
Hi,
Now,I am trying to build a Bloom Filter(BF) before join. The BF is
build
in
each node and sent to one node to combine. I want to set a stop sign
there
before sending the BF in each node. The stop sign means it can only
send
the BF after it is build.
The class HyracksConnection.waitForCompletion may help this. But I am
not
sure how to use it.
Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
jobidSendBF=hcc.startJob(); ?
Has anyone ever used the HyracksConnection.waitForCompletion?
Thanks,
Mingda
