Hi, Mingda.
What you need is quite familiar with what I and Presten have done.
Actually, I think we just need a shared
object accommodated by joblet or task which should be also driven by a
broadcast connector inbetween its input
and output operators. We can talk about this by skype if needed.
Best, Wenhai
> -----原始邮件-----
> 发件人: "Mike Carey" <dtab...@gmail.com>
> 发送时间: 2016年10月11日 星期二
> 收件人: dev@asterixdb.apache.org
> 抄送:
> 主题: Re: Let one Operator finished the job before another one begin in Hyracks
>
> And both Wenhai and Preston have examples of doing the
> fan-in-and-compute/fan-back-out pattern with blocking until the latter
> part is done - Wenhai for finding range split points for parallel
> sorting and Preston for similar things that arise in interval joins.
> Can you guys chime in when you have a chance? (Preston may be busy from
> what I saw on Skype on Friday :-), with congrats being due!)
>
>
> On 10/11/16 12:22 AM, Jianfeng Jia wrote:
> > Based on the described example, it seems possible to implement it in one
> > job by using MToNPartitioningConnectorDescriptor.
> > You can force that merge-BF-operator only runs in one partition by using
> > PartitionConstraintHelper.addAbsoluteLocationConstraint() function.
> >
> >> On Oct 10, 2016, at 11:43 PM, mingda li <limingda1...@gmail.com> wrote:
> >>
> >> Yeah, that will be easier. But for example, we have N nodes and in each
> >> node, it will generate a Bloom Filter(BF) for its own data. We need to send
> >> these BFs to one node for constructing a complete BF and then send the BF
> >> back to each node. I am not sure we can use multiple stage job for this,
> >> because there should be a 1->N and a N->1 connecter among nodes. If in one
> >> job, there may be no way to transfer data among nodes.
> >> This is my idea. If this can be implemented by one multiple stage job, that
> >> will decrease a lot of my work :-)
> >>
> >> Bests,
> >> Mingda
> >>
> >> On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <dtab...@gmail.com> wrote:
> >>
> >>> Is there a reason for wanting two jobs? I would think that one multiple
> >>> stage job would be preferable.
> >>>
> >>> On Oct 10, 2016 1:21 PM, "mingda li" <limingda1...@gmail.com> wrote:
> >>>
> >>>> Oh, thanks Kim~
> >>>>
> >>>> On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> >>>>
> >>>>> Forwarded to dev.
> >>>>>
> >>>>> Best,
> >>>>> Taewoo
> >>>>>
> >>>>> ---------- Forwarded message ----------
> >>>>> From: mingda li <limingda1...@gmail.com>
> >>>>> Date: Mon, Oct 10, 2016 at 11:21 AM
> >>>>> Subject: Let one Operator finished the job before another one begin in
> >>>>> Hyracks
> >>>>> To: us...@asterixdb.apache.org
> >>>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> Now,I am trying to build a Bloom Filter(BF) before join. The BF is
> >>> build
> >>>> in
> >>>>> each node and sent to one node to combine. I want to set a stop sign
> >>>> there
> >>>>> before sending the BF in each node. The stop sign means it can only
> >>> send
> >>>>> the BF after it is build.
> >>>>> The class HyracksConnection.waitForCompletion may help this. But I am
> >>>> not
> >>>>> sure how to use it.
> >>>>> Should I build two jobs: hcc.waitForCompletion(jobBuildBF);
> >>>>> jobidSendBF=hcc.startJob(); ?
> >>>>> Has anyone ever used the HyracksConnection.waitForCompletion?
> >>>>>
> >>>>> Thanks,
> >>>>> Mingda
> >>>>>
>
