Ok, I got it. our data delivery channel works in GMT+4, so saving all GMT+4 data in UTF would be more confusing. Imagine pain of analysits when they''ll start to work with data partitions... Thanks, now it's clear to me.
2013/10/16 Mohammad Islam <[email protected]> > Hi, > > Please think everything from UTC point of view. > Your data set directory should be based on UTC as well. > Only conversion that you will do when you specify the "start-time" which > should be your start time time (based on your TZ) - 4 hours. > > Now think from UTC perspective. > For example, coordinator action at 7:00 UTC (11:00 your time) should use > 6:00 UTC data as current(-1). > > I would request not to use current(-3). Again data directory should be > based on UTC as well. > > I know these are confusing sometime. > > Regards, > Mohammad > > > > > > On Sunday, October 13, 2013 6:09 AM, Serega Sheypak < > [email protected]> wrote: > > I've changed > > <input-events> > <data-in name="action_stg_in" dataset="action_stg"> > <instance>*${coord:current(-1)}*</instance> > </data-in> > </input-events> > > <output-events> > <data-out name="action_md_out" dataset="action_md"> > <instance>*${coord:current(-1)}*</instance> > </data-out> > </output-events> > > to > > <input-events> > <data-in name="action_stg_in" dataset="action_stg"> > <instance>*${coord:current(3)}*</instance> > > </data-in> > </input-events> > > <output-events> > <data-out name="action_md_out" dataset="action_md"> > <instance>*${coord:current(3)}*</instance> > </data-out> > </output-events> > > Now it works as expected, but this > *${coord:current(3)} looks like magic number... Why 3, not -1 or 123? > Because server has GMT+04:00 timezone?* > > > > 2013/10/13 Serega Sheypak <[email protected]> > > > Thank you very much. I did it the way you say, now I have other problem. > ^( > > > > I want my coordinator to run each hour and take data from previous hour. > > > > The problem is that hour difference =5 hours. > > > > Imagine current time is 14:31 > > My coordinator materializes each hour on third minute and takes dataset > > from previous hour (some/hdfs/path/2103/10/12/*13*) > > > > But it materalized for hour 09 and does nothing else. I do understand why > > difference is 5 hours = GMT+04:00 + 1 hour from corodinator settings > (take > > previous hour) > > > > I don't want 5 hour lag, I want 1 hour lag.What do I do wrong? > > > > The settings fot timezones are > UTC: > > > > The part of coorinator is: > > > > <datasets> > > <dataset name="action_stg" > > frequency="${coord:hours(1)}" > > initial-instance="2013-01-01T00:02Z" > > timezone="${timezone}"> > > > > > <uri-template>${nameNode}/staging/landing/stream/action/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template> > > <done-flag></done-flag> > > </dataset> > > <dataset name="action_md" > > frequency="${coord:hours(1)}" > > initial-instance="2013-01-01T00:02Z" > > timezone="${timezone}"> > > > > > <uri-template>${nameNode}/masterdata/stream/action/${YEAR}/${MONTH}/${DAY}/${HOUR}</uri-template> > > <done-flag></done-flag> > > </dataset> > > </datasets> > > > > <input-events> > > <data-in name="action_stg_in" dataset="action_stg"> > > > <instance>${coord:current(-1)}</instance> > > </data-in> > > </input-events> > > > > <output-events> > > <data-out name="action_md_out" dataset="action_md"> > > <instance>${coord:current(-1)}</instance> > > </data-out> > > </output-events> > > > > > > > > > > > > > > 2013/10/13 Mohammad Islam <[email protected]> > > > >> Fully agreed with Robert. > >> Reiterating .. > >> Please use UTC in all cases. And define all time values (such as start > >> time, end time , initial time) based on UTC. For your example, 00:03 > local > >> time is previous day 20:03 UTC time. > >> > >> "timezone"parameter doesn't have any impact in this case -- only for > DST. > >> > >> Regards, > >> Mohammad > >> > >> > >> > >> On Friday, October 11, 2013 6:21 PM, Robert Kanter < > [email protected]> > >> wrote: > >> > >> Hi Serega, > >> > >> Oozie always runs everything in > "oozie.processing.timezone", which > >> defaults > >> to UTC. The "timezone" in the coordinator is a little misleading as it > >> doesn't actually change the timezone; only the daylight savings time > rules > >> from this timezone are used. > >> > >> We typically recommend users to leave the "oozie.processing.timezone" at > >> UTC and to do the math for setting the times in your coordinator (like > >> Mohammad said). > >> > >> > >> - Robert > >> > >> > >> > >> On Thu, Oct 10, 2013 at 2:56 AM, Serega Sheypak < > [email protected] > >> >wrote: > >> > >> > Yesterday I've set > >> > oozie.processing.timezone = UTC (back, it's default value) > >> > and set startTime=20:03 of previous day. > >> > timezone for coordinator was: GMT+0400, GMT+0000, UTC > >> > > >> > Yes. It works. It really starts as 00:03 of the next day. > >> > But really "timezone" value (GMT+0400, GMT+0000, UTC) didn't change > >> > something. Oozie always adds +4 hours. > >> > > >> > Can you please explain this trick? > >> > I just want to run coordinator each day at 00:03 why do I have to > >> perform > >> > such non evident hour calculations for each coordinator? > >> > > >> > Please explain. I don't understand the reason of complicity. Thank > you. > >> > > >> > > >> > > >> > 2013/10/10 Mohammad Islam <[email protected]> > >> > > >> > > Can you please set two timezone values ( oozie.processing.timezone > >> > > and timezone) to UTC? > >> > > Basically don't specify any timezone value other than UTC. > >> > > > >> > > Then try with the following start times : startTime=20:03 <previous > >> > > days>/startTime=00:03/startTime=04:03. > >> > > Please let me know what you > got from this three cases. > >> > > > >> > > After getting your response, i can explain to you (if needed). > >> > > > >> > > Regards, > >> > > Mohammad > >> > > > >> > > > >> > > > >> > > > >> > > ________________________________ > >> > > From: Serega Sheypak <[email protected]> > >> > > To: [email protected] > >> > > Sent: Wednesday, October 9, 2013 12:06 PM > >> > > Subject: > HowTo make Oozie respect timezone? > >> > > > >> > > > >> > > Hi, I completely don't understand how timezones work in oozie. > >> > > Oozie server is in GMT+0400 > >> > > Oozie timezone property: oozie.processing.timezone=UTC > >> > > > >> > > I want to run daily coordinator at 00:02 each day > >> > > What do I have to set in coordinator fields? > >> > > > >> > > Right know I try to set > >> > > startTime=00:03 > >> > > timeZone=GMT+0400 > >> > > > >> > > And Oozie starts coordinator at 04:03. > >> > > I've tried t set timezone of coordinator to UTC, GMT-0400, nothing > >> helps. > >> > > > >> > > Anyway oozie starts coordinator at 04:03 > >> > > What do I do wrong? > >> > > > >> > > >> > > > > >
