I guess the HCAT integration (in hive and oozie, both WIP) would be the answer for that.
On Wed, Feb 13, 2013 at 10:04 AM, Harish Krishnan < [email protected]> wrote: > I have also faced similar issue. Would appreciate a solution for this > issue. > > Thanks & Regards, > Harish.T.K > > > On Wed, Feb 13, 2013 at 1:39 AM, Mohammad Islam <[email protected]> > wrote: > > > For data checking , did you consider coordinator in place of custom > logic? > > > > Do you see any issue with external table in this respect? > > > > Regards, > > Mohammad > > > > > > ________________________________ > > From: Shreehari Padaki <[email protected]> > > To: [email protected]; Mohammad Islam <[email protected]> > > Cc: Harsh J <[email protected]> > > Sent: Wednesday, February 13, 2013 12:13 AM > > Subject: Re: Oozie - accessing files from local file system > > > > yes, we can do that way but it adds one more layer and also requires some > > extra time. > > > > Another thing is i want to schedule my hive queries once data is loaded > in > > hive tables. So now we are creating external table (to hive) in hdfs and > > loading files through cron job and using some custom logic in oozie to > > check data is uploaded in hdfs or not before it runs hive queries. > > > > So just wanted to check for other ways to do that. > > > > Thanks > > Shreehari > > > > > > On Wed, Feb 13, 2013 at 1:34 PM, Mohammad Islam <[email protected]> > > wrote: > > > > > > > > > > > Hi Shreehari, > > > I think there is no way of executing this at this point. > > > You are right, Oozie needs the file to be in hdfs. > > > Is there a way of uploading the files in hdfs and then execute with > LOAD > > > command giving the hdfs path as the source. > > > > > > Regards, > > > Mohammad > > > > > > > > > ________________________________ > > > From: Shreehari Padaki <[email protected]> > > > To: Harsh J <[email protected]> > > > Cc: [email protected] > > > Sent: Tuesday, February 12, 2013 11:53 PM > > > Subject: Re: Oozie - accessing files from local file system > > > > > > Hi Harsh, > > > > > > We have tried giving absolute path but it didn't work. It always try to > > > look for the path in hdfs. > > > > > > So what is the way to schedule a job to copy files from local file > system > > > to hdfs or hive? is this possible through oozie? > > > > > > > > > Thanks > > > Shreehari > > > > > > On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <[email protected]> wrote: > > > > > > > Hi Shreehari, > > > > > > > > I'd recommend using a list since my limited free time does not permit > > > > me to make sure to reply to every single email sent to me personally. > > > > Using a community mailing list also lets you get thoughts of several > > > > experienced folks out there, than just one person :) > > > > > > > > I'd recommend sending the question to the Oozie lists, > > > > [email protected]. I've added the list to this response. You can > > > > subscribe to these lists via instructions at > > > > http://oozie.apache.org/mail-lists.html. > > > > > > > > From a brief read, however, your problem is that Oozie may not > > > > understand relative paths, given the user/location it runs that > > > > command as. Try passing an absolute path, such as > > > > "/user/shreehari/downloads/test/" for example, and verify? > > > > > > > > On Wed, Feb 13, 2013 at 10:59 AM, <[email protected]> wrote: > > > > > Hi Harsha, > > > > > > > > > > We are planning to use Oozie as work flow scheduler with Hadoop and > > > Hive. > > > > > > > > > > But the problem we are facing is we are unable to access files > stored > > > in > > > > local file system to upload it in hive through Oozie. > > > > > > > > > > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST; > > > > > > > > > > if i execute this in hive then it works fine, but same thing if run > > it > > > > through Oozie it throws error saying the path 'downloads/test/' as > > > invalid > > > > path. > > > > > > > > > > As i understood Oozie run over hdfs, so it is expecting files in > > hdfs. > > > > > > > > > > I am new to hadoop and oozie. Is there anyway we can achieve this > > > > through Oozie? > > > > > > > > > > is there any better work flow scheduler we can use? > > > > > > > > > > Thanks > > > > > Shreehari > > > > > > > > > > > > > > > > -- > > > > Harsh J > > > > > > > > > > -- Alejandro
