Support for EL function to append pathFilter to all the paths materialized for
coord:dataIn('inPath-event')
-----------------------------------------------------------------------------------------------------------
Key: OOZIE-699
URL: https://issues.apache.org/jira/browse/OOZIE-699
Project: Oozie
Issue Type: New Feature
Environment: Oozie 3.0.2, CDH3U0.
Reporter: Shaik Idris Ali
Hi currently, the coordinator does not support gating/waiting on URI with
patterns/filters.
My dataset in coordinator.xml is:
<datasets><dataset name="inPath" frequency="${coord:hours(1)}"
initial-instance="2012-01-30T00:00Z"
timezone="UTC"><uri-template>${nameNode}/projects/bi/ioout/${YEAR}-${MONTH}-${DAY}-${HOUR}</uri-template><done-flag></done-flag></dataset>
My input-event is:
<input-events><data-in name="inPath-event" dataset="inPath">
<start-instance>${ivory:now(-25,0)}</start-instance>
<end-instance>${ivory:now(0,0)}
</end-instance></data-in></input-events>
However I would like to pass, the materialized events with appended filter
(/clickon/*/*/OK/*/*) to workflow:
<property><name>inPath</name><value>${coord:dataIn('inPath-event')}/clickon/*/*/OK/*/*</value>
workflow can have actions actions as pig script or M/R Job.
The resolved URIs are:
hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-07,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-06,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-05,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-04,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-03,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-02,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-01,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-00/clickon/*/*/OK/*/*
i.e. the pathFilter or pattern is appended to only last URI.
It would be good if we have an coordinator EL function:
coord:appendPathFilter(coord:dataIn('inPath-event'),/clickon/*/*/OK/*/*)
This is a very common usecase while working with pig scripts or M/R jobs.
Thanks,
-Idris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
