Yes, please if you can open a bug JIRA that'd be great! On Wed, Jun 19, 2019 at 7:51 PM David Crespi < david.cre...@storedgesystems.com> wrote:
> Patrick, would you like me to open up a bug, or do you have this? > > > > Regards, > > > > David > > > > ________________________________ > From: Patrick Stuedi <pstu...@gmail.com> > Sent: Wednesday, June 19, 2019 6:40:10 AM > To: dev@crail.apache.org > Cc: Jonas Pfefferle; d...@crail.incubator.apache.org > Subject: Re: Crail used as type 2 storage for TeraSort does not catch the > "finished" signal > > Ok. The closing issue you reported is a bug I think. And the changes of > Adrian are targeting the shuffle plugin only, so they're not fixing the > closing issue. We need to look into this more carefully, right now there is > no easy fix I guess, sorry. > > -Patrick > > On Wed, Jun 19, 2019 at 3:32 PM David Crespi < > david.cre...@storedgesystems.com> wrote: > > > Hi Patrick, I am attempting to use the crail shuffle plugin… different > > thread. > > > > This was just about using crail for input/output. I have HDFS set up > along > > > > side, to also benchmark the different configs. > > > > > > > > Regards, > > > > > > > > David > > > > > > > > > > > > ________________________________ > > From: Patrick Stuedi <pstu...@gmail.com> > > Sent: Wednesday, June 19, 2019 6:29:11 AM > > To: dev@crail.apache.org > > Cc: Jonas Pfefferle; d...@crail.incubator.apache.org > > Subject: Re: Crail used as type 2 storage for TeraSort does not catch the > > "finished" signal > > > > The closing issue is related to using Crail for input/output. The changes > > Adrian made just earlier today are changes on the shuffle plugin. Are you > > using the Crail shuffle plugin at all? If not then the changes of Adrian > > are not relevant to you. > > > > -Patrick > > > > On Wed, Jun 19, 2019 at 3:22 PM David Crespi < > > david.cre...@storedgesystems.com> wrote: > > > > > Thanks for the description. It seemed odd that it behaved this way as > > > HDFS does close as > > > > > > expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark > > > numbers? > > > > > > > > > > > > Regards, > > > > > > > > > > > > David > > > > > > > > > > > > C: 714-476-2692 > > > > > > > > > > > > ________________________________ > > > From: Jonas Pfefferle <peppe...@japf.ch> > > > Sent: Wednesday, June 19, 2019 12:17:30 AM > > > To: dev@crail.apache.org; David Crespi; d...@crail.incubator.apache.org > > > Subject: Re: Crail used as type 2 storage for TeraSort does not catch > the > > > "finished" signal > > > > > > Hi David, > > > > > > > > > Unfortunately, if you use Crail for input/output with Spark this is > > > expected. The problem is Spark never closes the filesystem correctly. I > > > haven't look into this lately but if I remember correctly there was no > > easy > > > way otherwise to determine Spark is about to close. > > > > > > Regards, > > > Jonas > > > > > > On Tue, 18 Jun 2019 22:17:16 +0000 > > > David Crespi <david.cre...@storedgesystems.com> wrote: > > > > Hi, > > > > I’m running Crail as the temporary backend storage for Terasort. > > > > After each section (TeraGen, TeraSort, TeraVerify) > > > > the program waits until a Cntl-C is given, then moves on to the next > > > >section. Is this the expected behavior, or is > > > > this a bug? > > > > > > > > Here’s a small snippet of the output. Terasort waits where the > > > >bolded “Number of records” is listed, until > > > > The ^c is given. Each of the three programs does the same, but the > > > >program does finish without errors. > > > > > > > > > > > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name: > > > >TaskSet_1.0, runningTasks: 1 > > > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage > > > >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2) > > > > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in > > > >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB) > > > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name: > > > >TaskSet_1.0, runningTasks: 0 > > > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage > > > >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2) > > > > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose > > > >tasks have all completed, from pool > > > > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at > > > >TeraGen.scala:94) finished in 0.995 s > > > > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1, > > > >remaining stages = 0 > > > > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at > > > >TeraGen.scala:94, took 1.003537 s > > > > Number of records written: 10000 > > > > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown > > > >hook > > > > > > > > Regards, > > > > > > > > David > > > > > > > > > > > > > > > > >
