There is no need to configure anything extra with the proposed change, it just brings back LM to how it worked before.
There is no point modifying n tests for extra setup with no gain. Thomas On Thu, Sep 3, 2015 at 9:14 AM, Chetan Narsude <[email protected]> wrote: > Why does it matter that AsyncFSStorageAgent is being used with > LocalCluster? It using the localfs and hence no gain is the implementation > detail that's abstracted out by FileSystem already. > > If there is a problem of random artifacts left behind after the test, there > is a reason and most likely it's misconfiguration of the StorageAgent. Why > wouldn't that be fixed. > > -- > Chetan > > > On Thu, Sep 3, 2015 at 8:59 AM, Amol Kekre <[email protected]> wrote: > > > Clean up container files left over should be a distributed OS task. Clean > > up, back up, archive, ... all is for the OS (aka YARN). We must assume > kill > > -9. > > > > The only thing where the operator comes into play is "teardown()", which > is > > business logic (not Apex engine) issue. This could be db connection etc. > > > > Thks, > > Amol > > > > On Thu, Sep 3, 2015 at 8:52 AM, Thomas Weise <[email protected]> > > wrote: > > > > > When the container gets killed, we should not assume anything about > > > cleanup. It can be a kill -9. Any related "cleanup" falls under nice to > > > have, no guarantees. > > > > > > On Thu, Sep 3, 2015 at 8:49 AM, Chandni Singh <[email protected] > > > > > wrote: > > > > > > > I have a question regarding what Gaurav mentioned > > > > ---- > > > > When container runs in cluster, "." specifies the containers local > path > > > on > > > > the node where container specific jars and other resources resides. > It > > > > creates a folder under that which is live as long as container lives. > > So > > > > there are no vagrant folders anywhere > > > > --- > > > > > > > > When the container gets killed, do we cleanup the folders created by > > > Async > > > > under the containers working dir? > > > > > > > > On Thu, Sep 3, 2015 at 8:42 AM, Thomas Weise <[email protected] > > > > > > wrote: > > > > > > > >> It makes sense to use the synchronous checkpointing for the local > > mode. > > > >> LM is meant to simplify dependencies and setup. The default for > > > execution > > > >> on YARN remains async. > > > >> > > > >> Thomas > > > >> > > > >> > > > >> On Thu, Sep 3, 2015 at 8:34 AM, Chandni Singh < > > [email protected]> > > > >> wrote: > > > >> > > > >>> APPLICATION_PATH isn't related to local base dir of Async as far > as I > > > >>> know. StramLocalCluster sets the APP_PATH to "target/...". > > > >>> StramLocalCluster should use FSStorageAgent. > > > >>> > > > >>> - Chandni > > > >>> > > > >>> On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta < > [email protected] > > > > > > >>> wrote: > > > >>> > > > >>>> As Thomas mentioned as default remains to be async. You can either > > > >>>> change the storage agent or set the APPLICATION_PATH. > > > >>>> > > > >>>> When container runs in cluster, "." specifies the containers local > > > path > > > >>>> on the node where container specific jars and other resources > > > resides. It > > > >>>> creates a folder under that which is live as long as container > > lives. > > > So > > > >>>> there are no vagrant folders anywhere > > > >>>> > > > >>>> Thanks > > > >>>> -Gaurav > > > >>>> > > > >>>> On Wed, Sep 2, 2015 at 11:33 PM, Chandni Singh < > > > [email protected] > > > >>>> > wrote: > > > >>>> > > > >>>>> I think there is a problem in the default Async as well. It also > > uses > > > >>>>> the working directory as its local base path. > > > >>>>> > > > >>>>> In the Async -> copyToHdfs() method, we delete the window files > > but > > > >>>>> the folder with the operator name never gets deleted. > > > >>>>> So on the cluster there will be such vagrant folders in the > > working > > > >>>>> directory? > > > >>>>> > > > >>>>> On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise < > > > [email protected]> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Chandni, > > > >>>>>> > > > >>>>>> Agreed. See whether the tests work with the synchronous storage > > > >>>>>> agent. If yes, change them. The default needs to remain async. > > > >>>>>> > > > >>>>>> Thomas > > > >>>>>> > > > >>>>>> > > > >>>>>> On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh < > > > >>>>>> [email protected]> wrote: > > > >>>>>> > > > >>>>>>> Hi, > > > >>>>>>> > > > >>>>>>> I would like to know what was the reason to use > > AsyncFSStorageAgent > > > >>>>>>> with StramLocalCluster? > > > >>>>>>> StramLocalCluster is mainly for testing in a non-distributed > mode > > > >>>>>>> and I am unclear how AsyncFSStorageAgent is helpful in this > mode. > > > >>>>>>> > > > >>>>>>> Thanks, > > > >>>>>>> Chandni > > > >>>>>>> > > > >>>>>>> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh < > > > >>>>>>> [email protected]> wrote: > > > >>>>>>> > > > >>>>>>>> This is because of recent changes to StramLocalCluster where > > > >>>>>>>> AsyncFSStorageAgent is used for checkpointing > > > >>>>>>>> > > > >>>>>>>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new > > > AsyncFSStorageAgent(new Path(pathUri, > > > LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null)); > > > >>>>>>>> > > > >>>>>>>> The AsyncFSStorageAgent(String path, Configuration conf) uses > > "." > > > as localBasePath and therefore creates sub-directories per operator in > > the > > > current working directory. > > > >>>>>>>> > > > >>>>>>>> I am going to create a ticket to address this and will fix it. > > > >>>>>>>> > > > >>>>>>>> -Chandni > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh < > > > >>>>>>>> [email protected]> wrote: > > > >>>>>>>> > > > >>>>>>>>> Hi, > > > >>>>>>>>> > > > >>>>>>>>> I can see empty folders getting created under Malhar/lib > called > > > >>>>>>>>> '1' and '2'. > > > >>>>>>>>> I think this is because of using LocalMode to run a test > > > >>>>>>>>> application. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> If anyone has checked in such cases please do check and let > us > > > >>>>>>>>> know. > > > >>>>>>>>> > > > >>>>>>>>> Thanks, > > > >>>>>>>>> Chandni > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > >
