APPLICATION_PATH isn't related to local base dir of Async as far as I know. StramLocalCluster sets the APP_PATH to "target/...". StramLocalCluster should use FSStorageAgent.
- Chandni On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta <[email protected]> wrote: > As Thomas mentioned as default remains to be async. You can either change > the storage agent or set the APPLICATION_PATH. > > When container runs in cluster, "." specifies the containers local path on > the node where container specific jars and other resources resides. It > creates a folder under that which is live as long as container lives. So > there are no vagrant folders anywhere > > Thanks > -Gaurav > > On Wed, Sep 2, 2015 at 11:33 PM, Chandni Singh <[email protected]> > wrote: > >> I think there is a problem in the default Async as well. It also uses the >> working directory as its local base path. >> >> In the Async -> copyToHdfs() method, we delete the window files but the >> folder with the operator name never gets deleted. >> So on the cluster there will be such vagrant folders in the working >> directory? >> >> On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise <[email protected]> >> wrote: >> >>> Chandni, >>> >>> Agreed. See whether the tests work with the synchronous storage agent. >>> If yes, change them. The default needs to remain async. >>> >>> Thomas >>> >>> >>> On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I would like to know what was the reason to use AsyncFSStorageAgent >>>> with StramLocalCluster? >>>> StramLocalCluster is mainly for testing in a non-distributed mode and I >>>> am unclear how AsyncFSStorageAgent is helpful in this mode. >>>> >>>> Thanks, >>>> Chandni >>>> >>>> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh <[email protected] >>>> > wrote: >>>> >>>>> This is because of recent changes to StramLocalCluster where >>>>> AsyncFSStorageAgent is used for checkpointing >>>>> >>>>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new >>>>> AsyncFSStorageAgent(new Path(pathUri, >>>>> LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null)); >>>>> >>>>> The AsyncFSStorageAgent(String path, Configuration conf) uses "." as >>>>> localBasePath and therefore creates sub-directories per operator in the >>>>> current working directory. >>>>> >>>>> I am going to create a ticket to address this and will fix it. >>>>> >>>>> -Chandni >>>>> >>>>> >>>>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I can see empty folders getting created under Malhar/lib called '1' >>>>>> and '2'. >>>>>> I think this is because of using LocalMode to run a test application. >>>>>> >>>>>> >>>>>> If anyone has checked in such cases please do check and let us know. >>>>>> >>>>>> Thanks, >>>>>> Chandni >>>>>> >>>>> >>>>> >>>> >>> >> >
