> On March 5, 2014, 6:03 p.m., Purshotam Shah wrote: > > core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java, line 962 > > <https://reviews.apache.org/r/18762/diff/1/?file=510568#file510568line962> > > > > I see some issue with this. Earlier we start from end and break the for > > loop if nominalInstanceCal.compareTo(initInstance). > > There are still some instance being return, but not in this case. > > > > Eg > > <input-events> > > <data-in name="input" dataset="raw-logs"> > > <start-instance>${coord:current(-200)}</start-instance> > > <end-instance>${coord:current(0)}</end-instance> > > </data-in> > > </input-events> > > > > Where coord:current(-10) < initInstance. > > > > Earlier it will return -9 to 0, but now none. > > > > > > > > shwethags wrote: > Actually, instead of returning partial instances and user wondering what > went wrong with this coord action, we should add this check at coord submit. > Coord submit should fail if the instances span before/after dataset time > interval
No. User cloud be using this. User might want to process full data + increment data every time. They will set start-instance as coord:current(-1000000) (-very high number), and end-instance as current(0), so that they can process full feed + incremental feed. This cloud be one way to process all data from start date to current(0). - Purshotam ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18762/#review36237 ----------------------------------------------------------- On March 5, 2014, 5:28 a.m., shwethags wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18762/ > ----------------------------------------------------------- > > (Updated March 5, 2014, 5:28 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-1709 > https://issues.apache.org/jira/browse/OOZIE-1709 > > > Repository: oozie-git > > > Description > ------- > > CoordELFunctions.getCurrentInstance() has this code: > while (current.compareTo(calEffectiveTime) <= 0) { > current = (Calendar) origCurrent.clone(); > instanceCount[0]++; > current.add(dsTimeUnit.getCalendarUnit(), instanceCount[0] * > dsFreq); > } > > For coords with smaller frequency and start time in very past, this is very > expensive. On prod, we have seen materialisation of each instance taking few > mins sometimes for coords with 1 min frequency > > > Diffs > ----- > > core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java d73bc7d > core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 34a428f > > Diff: https://reviews.apache.org/r/18762/diff/ > > > Testing > ------- > > UT > > > Thanks, > > shwethags > >
