Samarth Gupta created FALCON-494:
------------------------------------
Summary: update feature broken
Key: FALCON-494
URL: https://issues.apache.org/jira/browse/FALCON-494
Project: Falcon
Issue Type: Bug
Components: general
Affects Versions: 0.6
Environment: QA
Reporter: Samarth Gupta
Priority: Blocker
Fix For: 0.5, 0.4
when we update an process / feed , the update take place from current time, and
all the future instance of the old coordinator are moved to new coord with new
definition.
However with the latest falcon, the coord which is getting created is having
all instances which have have been already processed by old coodinator.
Resulting in rerunning on those instances which are already in RUNNING / READY
/ WAITING state in old coord.
in the logs, following lines can be seen which says the new start time :
{code}
2014-06-25 06:36:40,819 INFO -
[1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Submitted
0000328-140624044727425-oozie-oozi-B on cluster
corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e
with properties: {nameNode=hdfs://192.168.138.182:8020,
oozie.use.system.libpath=true, field2=value2, jobTracker=192.168.138.182:8021,
field1=value1, falcon.libpath=/projectsTest/ivory/working/lib, colo.name=ua3,
oozie.bundle.application.path=${nameNode}/projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513,
user.name=dataqa} (OozieWorkflowEngine:1316)
2014-06-25 06:36:40,827 INFO -
[1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ New bundle
0000328-140624044727425-oozie-oozi-B scheduled successfully with start time
2014-06-25T06:39Z (OozieWorkflowEngine:1131)
2014-06-25 06:36:40,828 DEBUG -
[1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Updating endtime of coord
0000323-140624044727425-oozie-oozi-C to 2014-06-25T06:39Z on cluster
corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e
(OozieWorkflowEngine:1055)
{code}
but in the coord xml the start time is till the original process time :
{code}
2014-06-25 06:36:39,925 DEBUG -
[1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ Writing definition to
/projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513/DEFAULT/coordinator.xml
on cluster
corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e
(OozieWorkflowBuilder:338)
2014-06-25 06:36:39,925 DEBUG -
[1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395
1118a58a-4a4e-4052-afea-40d4960470f3] ~ <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
<coordinator-app
name="FALCON_PROCESS_DEFAULT_agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395"
frequency="${coord:minutes(1)}" start="2014-06-25T06:34Z"
end="2014-06-25T06:44Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.3">
<controls>
<timeout>30</timeout>
<concurrency>1</concurrency>
<execution>FIFO</execution>
<throttle>60</throttle>
</controls>
<datasets>
<dataset name="inputData" frequency="${coord:minutes(1)}"
initial-instance="2009-02-01T00:00Z" timezone="UTC">
<uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
<done-flag></done-flag>
</dataset>
<dataset name="inputData2" frequency="${coord:minutes(1)}"
initial-instance="2009-02-01T00:00Z" timezone="UTC">
<uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
<done-flag></done-flag>
</dataset>
{code}
most of the regression NewPrismProcessUpdateTest is failing coz of this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
