I have a question regarding what Gaurav mentioned ---- When container runs in cluster, "." specifies the containers local path on the node where container specific jars and other resources resides. It creates a folder under that which is live as long as container lives. So there are no vagrant folders anywhere ---
When the container gets killed, do we cleanup the folders created by Async under the containers working dir? On Thu, Sep 3, 2015 at 8:42 AM, Thomas Weise <[email protected]> wrote: > It makes sense to use the synchronous checkpointing for the local mode. LM > is meant to simplify dependencies and setup. The default for execution on > YARN remains async. > > Thomas > > > On Thu, Sep 3, 2015 at 8:34 AM, Chandni Singh <[email protected]> > wrote: > >> APPLICATION_PATH isn't related to local base dir of Async as far as I >> know. StramLocalCluster sets the APP_PATH to "target/...". >> StramLocalCluster should use FSStorageAgent. >> >> - Chandni >> >> On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta <[email protected]> >> wrote: >> >>> As Thomas mentioned as default remains to be async. You can either >>> change the storage agent or set the APPLICATION_PATH. >>> >>> When container runs in cluster, "." specifies the containers local path >>> on the node where container specific jars and other resources resides. It >>> creates a folder under that which is live as long as container lives. So >>> there are no vagrant folders anywhere >>> >>> Thanks >>> -Gaurav >>> >>> On Wed, Sep 2, 2015 at 11:33 PM, Chandni Singh <[email protected]> >>> wrote: >>> >>>> I think there is a problem in the default Async as well. It also uses >>>> the working directory as its local base path. >>>> >>>> In the Async -> copyToHdfs() method, we delete the window files but >>>> the folder with the operator name never gets deleted. >>>> So on the cluster there will be such vagrant folders in the working >>>> directory? >>>> >>>> On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise <[email protected]> >>>> wrote: >>>> >>>>> Chandni, >>>>> >>>>> Agreed. See whether the tests work with the synchronous storage agent. >>>>> If yes, change them. The default needs to remain async. >>>>> >>>>> Thomas >>>>> >>>>> >>>>> On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I would like to know what was the reason to use AsyncFSStorageAgent >>>>>> with StramLocalCluster? >>>>>> StramLocalCluster is mainly for testing in a non-distributed mode and >>>>>> I am unclear how AsyncFSStorageAgent is helpful in this mode. >>>>>> >>>>>> Thanks, >>>>>> Chandni >>>>>> >>>>>> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> This is because of recent changes to StramLocalCluster where >>>>>>> AsyncFSStorageAgent is used for checkpointing >>>>>>> >>>>>>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new >>>>>>> AsyncFSStorageAgent(new Path(pathUri, >>>>>>> LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null)); >>>>>>> >>>>>>> The AsyncFSStorageAgent(String path, Configuration conf) uses "." as >>>>>>> localBasePath and therefore creates sub-directories per operator in the >>>>>>> current working directory. >>>>>>> >>>>>>> I am going to create a ticket to address this and will fix it. >>>>>>> >>>>>>> -Chandni >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I can see empty folders getting created under Malhar/lib called '1' >>>>>>>> and '2'. >>>>>>>> I think this is because of using LocalMode to run a test >>>>>>>> application. >>>>>>>> >>>>>>>> >>>>>>>> If anyone has checked in such cases please do check and let us know. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Chandni >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
