I think there is a problem in the default Async as well. It also uses the working directory as its local base path.
In the Async -> copyToHdfs() method, we delete the window files but the folder with the operator name never gets deleted. So on the cluster there will be such vagrant folders in the working directory? On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise <[email protected]> wrote: > Chandni, > > Agreed. See whether the tests work with the synchronous storage agent. If > yes, change them. The default needs to remain async. > > Thomas > > > On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh <[email protected]> > wrote: > >> Hi, >> >> I would like to know what was the reason to use AsyncFSStorageAgent with >> StramLocalCluster? >> StramLocalCluster is mainly for testing in a non-distributed mode and I >> am unclear how AsyncFSStorageAgent is helpful in this mode. >> >> Thanks, >> Chandni >> >> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh <[email protected]> >> wrote: >> >>> This is because of recent changes to StramLocalCluster where >>> AsyncFSStorageAgent is used for checkpointing >>> >>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new AsyncFSStorageAgent(new >>> Path(pathUri, LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null)); >>> >>> The AsyncFSStorageAgent(String path, Configuration conf) uses "." as >>> localBasePath and therefore creates sub-directories per operator in the >>> current working directory. >>> >>> I am going to create a ticket to address this and will fix it. >>> >>> -Chandni >>> >>> >>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I can see empty folders getting created under Malhar/lib called '1' and >>>> '2'. >>>> I think this is because of using LocalMode to run a test application. >>>> >>>> >>>> If anyone has checked in such cases please do check and let us know. >>>> >>>> Thanks, >>>> Chandni >>>> >>> >>> >> >
