With property file configuration, it's hard to say which directory is used for share lib.
Server deleting(or moving to “lost and found") may be risky. User may like to change their configuration. We have admin command to list sharelib directory. User can know all used sharelib directory(including property file configuration). May be we should document saying that user need to delete older sharelib while upgrading. Puru. From: Robert Kanter <[email protected]<mailto:[email protected]>> Date: Thursday, November 21, 2013 2:23 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Purshotam Shah <[email protected]<mailto:[email protected]>> Subject: Re: What happens to old sharelib when switching to the new timestamped one? Puru, what do you think about my “lost+found” idea above? I’m concerned that users who upgraded will be confused about what sharelib they are using. On Tue, Nov 19, 2013 at 10:30 AM, Robert Kanter <[email protected]<mailto:[email protected]>> wrote: The only concern I have about leaving them is that users could get confused about which sharelib they are actually using. They’re used to /user/oozie/share/lib/pig, and might think they’re using that when they are really using /user/oozie/share/lib/lib_<timestamp>/pig. What if we moved everything that doesn’t have a prefix into a “lost and found" folder instead of deleting it (e.g. /user/oozie/share/lib/lost+found)? This way it would be more obvious that Oozie is not using leftover directories/files but we wouldn’t be deleting them in case the user needs those files for something. thoughts? On Tue, Nov 19, 2013 at 10:08 AM, Purshotam Shah <[email protected]<mailto:[email protected]>> wrote: Robert, Yes, old sharelib will not be purged by oozie, we need to delete them manually. >Would it make sense to simply delete anything that doesn¹t have the ³lib_² >or ³launcher_² prefix? >The only issues I see with that are that (a) someone could have put other >files/dirs in share/lib and (b) it won¹t catch files/dirs that happen to >have those prefixes (e.g. ³share/lib/lib_mystuff²). I think those two >cases are ok because the user shouldn¹t be putting other stuff in >/user/oozie/share/lib anyway. Oozie server only maintains and purge lib_<timestamp> and launcher_<timestamp> directory. Others will be ignored. We don't know the reason of existing so I believe it's better to ignore them. Puru. On 11/18/13 5:44 PM, "Robert Kanter" <[email protected]<mailto:[email protected]>> wrote: >Hi, > >I was looking at the new sharelib stuff, and I¹m wondering what the story >is when switching from the old version to the new timestamped version. If >I had an existing sharelib from an older Oozie, then when I run the >oozie-setup.sh sharelib create tool, it results in this: > ># sudo -u oozie hadoop fs -ls share/lib > >Found 10 items > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:32 share/lib/distcp > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:32 share/lib/hcatalog > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:32 share/lib/hive > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:35 >share/lib/lib_20131118173502 > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:33 >share/lib/mapreduce-streaming > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:33 share/lib/oozie > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:33 share/lib/pig > >-rw-r--r-- 3 oozie oozie 1365 2013-11-18 17:33 >share/lib/sharelib.properties > >drwxr-xr-x - oozie oozie 0 2013-11-18 17:33 share/lib/sqoop > >I haven¹t tested it, but I assume that the old directories (e.g. >share/lib/pig), will never get cleaned up by Oozie because its only >looking >for the prefixes when purging, right? > >Would it make sense to simply delete anything that doesn¹t have the ³lib_² >or ³launcher_² prefix? >The only issues I see with that are that (a) someone could have put other >files/dirs in share/lib and (b) it won¹t catch files/dirs that happen to >have those prefixes (e.g. ³share/lib/lib_mystuff²). I think those two >cases are ok because the user shouldn¹t be putting other stuff in >/user/oozie/share/lib anyway. >Thoughts? > > >thanks >- Robert
