When the container gets killed, we should not assume anything about cleanup. It can be a kill -9. Any related "cleanup" falls under nice to have, no guarantees.
On Thu, Sep 3, 2015 at 8:49 AM, Chandni Singh <[email protected]> wrote: > I have a question regarding what Gaurav mentioned > ---- > When container runs in cluster, "." specifies the containers local path on > the node where container specific jars and other resources resides. It > creates a folder under that which is live as long as container lives. So > there are no vagrant folders anywhere > --- > > When the container gets killed, do we cleanup the folders created by Async > under the containers working dir? > > On Thu, Sep 3, 2015 at 8:42 AM, Thomas Weise <[email protected]> > wrote: > >> It makes sense to use the synchronous checkpointing for the local mode. >> LM is meant to simplify dependencies and setup. The default for execution >> on YARN remains async. >> >> Thomas >> >> >> On Thu, Sep 3, 2015 at 8:34 AM, Chandni Singh <[email protected]> >> wrote: >> >>> APPLICATION_PATH isn't related to local base dir of Async as far as I >>> know. StramLocalCluster sets the APP_PATH to "target/...". >>> StramLocalCluster should use FSStorageAgent. >>> >>> - Chandni >>> >>> On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta <[email protected]> >>> wrote: >>> >>>> As Thomas mentioned as default remains to be async. You can either >>>> change the storage agent or set the APPLICATION_PATH. >>>> >>>> When container runs in cluster, "." specifies the containers local path >>>> on the node where container specific jars and other resources resides. It >>>> creates a folder under that which is live as long as container lives. So >>>> there are no vagrant folders anywhere >>>> >>>> Thanks >>>> -Gaurav >>>> >>>> On Wed, Sep 2, 2015 at 11:33 PM, Chandni Singh <[email protected] >>>> > wrote: >>>> >>>>> I think there is a problem in the default Async as well. It also uses >>>>> the working directory as its local base path. >>>>> >>>>> In the Async -> copyToHdfs() method, we delete the window files but >>>>> the folder with the operator name never gets deleted. >>>>> So on the cluster there will be such vagrant folders in the working >>>>> directory? >>>>> >>>>> On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise <[email protected]> >>>>> wrote: >>>>> >>>>>> Chandni, >>>>>> >>>>>> Agreed. See whether the tests work with the synchronous storage >>>>>> agent. If yes, change them. The default needs to remain async. >>>>>> >>>>>> Thomas >>>>>> >>>>>> >>>>>> On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I would like to know what was the reason to use AsyncFSStorageAgent >>>>>>> with StramLocalCluster? >>>>>>> StramLocalCluster is mainly for testing in a non-distributed mode >>>>>>> and I am unclear how AsyncFSStorageAgent is helpful in this mode. >>>>>>> >>>>>>> Thanks, >>>>>>> Chandni >>>>>>> >>>>>>> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This is because of recent changes to StramLocalCluster where >>>>>>>> AsyncFSStorageAgent is used for checkpointing >>>>>>>> >>>>>>>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new >>>>>>>> AsyncFSStorageAgent(new Path(pathUri, >>>>>>>> LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null)); >>>>>>>> >>>>>>>> The AsyncFSStorageAgent(String path, Configuration conf) uses "." as >>>>>>>> localBasePath and therefore creates sub-directories per operator in >>>>>>>> the current working directory. >>>>>>>> >>>>>>>> I am going to create a ticket to address this and will fix it. >>>>>>>> >>>>>>>> -Chandni >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I can see empty folders getting created under Malhar/lib called >>>>>>>>> '1' and '2'. >>>>>>>>> I think this is because of using LocalMode to run a test >>>>>>>>> application. >>>>>>>>> >>>>>>>>> >>>>>>>>> If anyone has checked in such cases please do check and let us >>>>>>>>> know. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Chandni >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
