Oh, I see. Thanks :-)
On Tue, Oct 11, 2016 at 11:50 AM, Mike Carey <[email protected]> wrote: > I think there will also be some helpful design documents that he and > Preston can both share (with pics of the Hyracks jobs and the > activities/stages involved). > > > > > On 10/11/16 10:52 AM, mingda li wrote: > >> Oh, thanks for all the explanation:-) >> I will talk with Wenhai about how they implement such function and try to >> finish this in one job. >> >> Bests, >> Mingda >> >> On Tue, Oct 11, 2016 at 9:52 AM, 李文海 <[email protected]> wrote: >> >> >>> >>> -----原始邮件----- >>>> 发件人: "Yingyi Bu" <[email protected]> >>>> 发送时间: 2016年10月12日 星期三 >>>> 收件人: [email protected] >>>> 抄送: >>>> 主题: Re: Let one Operator finished the job before another one begin in >>>> >>> Hyracks >>> >>>> +1! >>>> >>>> Best, >>>> Yingyi >>>> >>>> On Tue, Oct 11, 2016 at 9:32 AM, Mike Carey <[email protected]> wrote: >>>> >>>> BUT AGAIN: I think the preferred solution in this case is to do it in >>>>> >>>> one >>> >>>> job. Mingda, I would suggest sync'ing up with Wenhai for a Skype >>>>> >>>> meeting >>> >>>> on how he/Preston have done essentially the very same thing in their >>>>> >>>> use >>> >>>> cases for parallel sorts and interval joins. Hyracks has everything >>>>> >>>> needed >>> >>>> for this, as it turns out, without a multi-job need. >>>>> >>>>> >>>>> >>>>> On 10/11/16 9:26 AM, Yingyi Bu wrote: >>>>> >>>>> You can search the usage of waitForCompletion in the code base, e.g.: >>>>>> >>>>>> APIFramework.java: >>>>>> >>>>>> public void executeJobArray(IHyracksClientConnection hcc, >>>>>> JobSpecification[] specs, PrintWriter out) >>>>>> throws Exception { >>>>>> for (JobSpecification spec : specs) { >>>>>> spec.setMaxReattempts(0); >>>>>> JobId jobId = hcc.startJob(spec); >>>>>> long startTime = System.currentTimeMillis(); >>>>>> hcc.waitForCompletion(jobId); >>>>>> long endTime = System.currentTimeMillis(); >>>>>> double duration = (endTime - startTime) / 1000.00; >>>>>> out.println("<pre>Duration: " + duration + " sec</pre>"); >>>>>> } >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> You start a job and get the job Id, and then you can wait on the job >>>>>> >>>>> id. >>> >>>> >>>>>> Best, >>>>>> >>>>>> Yingyi >>>>>> >>>>>> >>>>>> On Tue, Oct 11, 2016 at 1:45 AM, 李文海 <[email protected]> wrote: >>>>>> >>>>>> Hi, Mingda. >>>>>> >>>>>>> What you need is quite familiar with what I and Presten have >>>>>>> >>>>>> done. >>> >>>> Actually, I think we just need a shared >>>>>>> object accommodated by joblet or task which should be also driven by >>>>>>> >>>>>> a >>> >>>> broadcast connector inbetween its input >>>>>>> and output operators. We can talk about this by skype if needed. >>>>>>> Best, Wenhai >>>>>>> >>>>>>> >>>>>>> -----原始邮件----- >>>>>>> >>>>>>>> 发件人: "Mike Carey" <[email protected]> >>>>>>>> 发送时间: 2016年10月11日 星期二 >>>>>>>> 收件人: [email protected] >>>>>>>> 抄送: >>>>>>>> 主题: Re: Let one Operator finished the job before another one begin >>>>>>>> >>>>>>> in >>> >>>> Hyracks >>>>>>> >>>>>>> And both Wenhai and Preston have examples of doing the >>>>>>>> fan-in-and-compute/fan-back-out pattern with blocking until the >>>>>>>> >>>>>>> latter >>> >>>> part is done - Wenhai for finding range split points for parallel >>>>>>>> sorting and Preston for similar things that arise in interval joins. >>>>>>>> Can you guys chime in when you have a chance? (Preston may be busy >>>>>>>> >>>>>>> from >>> >>>> what I saw on Skype on Friday :-), with congrats being due!) >>>>>>>> >>>>>>>> >>>>>>>> On 10/11/16 12:22 AM, Jianfeng Jia wrote: >>>>>>>> >>>>>>>> Based on the described example, it seems possible to implement it >>>>>>>>> >>>>>>>> in >>> >>>> one job by using MToNPartitioningConnectorDescriptor. >>>>>>>> You can force that merge-BF-operator only runs in one partition by >>>>>>>> using PartitionConstraintHelper.addAbsoluteLocationConstraint() >>>>>>>> >>>>>>> function. >>>>>>> >>>>>>> On Oct 10, 2016, at 11:43 PM, mingda li <[email protected]> >>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>> Yeah, that will be easier. But for example, we have N nodes and in >>>>>>>> >>>>>>>>> each >>>>>>>>> >>>>>>>> node, it will generate a Bloom Filter(BF) for its own data. We need >>>>>>>> >>>>>>>>> to send >>>>>>>>> >>>>>>>> these BFs to one node for constructing a complete BF and then send >>>>>>>> >>>>>>>>> the BF >>>>>>>>> >>>>>>>> back to each node. I am not sure we can use multiple stage job for >>>>>>>> >>>>>>>>> this, >>>>>>>>> >>>>>>>> because there should be a 1->N and a N->1 connecter among nodes. If >>>>>>>> >>>>>>>>> in one >>>>>>>>> >>>>>>>> job, there may be no way to transfer data among nodes. >>>>>>>> >>>>>>>>> This is my idea. If this can be implemented by one multiple stage >>>>>>>>>> >>>>>>>>>> job, that >>>>>>>>> >>>>>>>> will decrease a lot of my work :-) >>>>>>>> >>>>>>>>> Bests, >>>>>>>>>> Mingda >>>>>>>>>> >>>>>>>>>> On Mon, Oct 10, 2016 at 8:59 PM, Mike Carey <[email protected]> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>> Is there a reason for wanting two jobs? I would think that one >>>>>>>> >>>>>>>>> multiple >>>>>>>>>> >>>>>>>>> stage job would be preferable. >>>>>>>> >>>>>>>>> On Oct 10, 2016 1:21 PM, "mingda li" <[email protected]> >>>>>>>>>>> >>>>>>>>>> wrote: >>> >>>> Oh, thanks Kim~ >>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 10, 2016 at 12:55 PM, Taewoo Kim < >>>>>>>>>>>> >>>>>>>>>>> [email protected]> >>> >>>> wrote: >>>>>>>>>>> >>>>>>>>>> Forwarded to dev. >>>>>>>> >>>>>>>>> Best, >>>>>>>>>>>>> Taewoo >>>>>>>>>>>>> >>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>> From: mingda li <[email protected]> >>>>>>>>>>>>> Date: Mon, Oct 10, 2016 at 11:21 AM >>>>>>>>>>>>> Subject: Let one Operator finished the job before another one >>>>>>>>>>>>> >>>>>>>>>>>>> begin in >>>>>>>>>>>> >>>>>>>>>>> Hyracks >>>>>>>> >>>>>>>>> To: [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Now,I am trying to build a Bloom Filter(BF) before join. The >>>>>>>>>>>>> >>>>>>>>>>>> BF is >>> >>>> build >>>>>>>>>>>> in >>>>>>>>>>>> >>>>>>>>>>>> each node and sent to one node to combine. I want to set a stop >>>>>>>>>>>>> >>>>>>>>>>>>> sign >>>>>>>>>>>> >>>>>>>>>>> there >>>>>>>> >>>>>>>>> before sending the BF in each node. The stop sign means it can >>>>>>>>>>>>> >>>>>>>>>>>> only >>> >>>> send >>>>>>>>>>>> the BF after it is build. >>>>>>>>>>>> >>>>>>>>>>>>> The class HyracksConnection.waitForCompletion may help this. >>>>>>>>>>>>> >>>>>>>>>>>> But >>> >>>> I am >>>>>>>>>>>> >>>>>>>>>>> not >>>>>>>> >>>>>>>>> sure how to use it. >>>>>>>>>>>>> Should I build two jobs: hcc.waitForCompletion(jobBuildBF); >>>>>>>>>>>>> jobidSendBF=hcc.startJob(); ? >>>>>>>>>>>>> Has anyone ever used the HyracksConnection.waitForCompletion? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Mingda >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>> >
