Hi, did anyone make it work property in his project?
I need to do dry run for my workflows.
The usecase is:
User writes workflow and wants to:
1. Check if it valid
2. do dryrun, see how it flows without executing steps.
Let say I have wflow with three steps:
1. disctp data from $A to $B
2. run spark action with $B as input
3. disctp $B to $C
I want to do dryrun and check how my variables were interpolated it wflow.
The killer feature is: I want to imitate spark action failure and check how
my kill node looks like.
I gave up to make MiniOozie work. But I was able to start DagEngine.
val fs: FileSystem = FileSystem.getLocal(new Configuration())
val appPath: Path = new Path("build", "app")
fs.mkdirs(appPath)
fs.mkdirs(new Path(appPath, "lib"))
val writer = new OutputStreamWriter(fs.create(new Path(appPath,
"workflow.xml")))
writer.write(pipelineXml)
writer.close()
val wc = getClient
val workflowConfiguration = wc.createConfiguration
workflowConfiguration.setProperty(OozieClient.APP_PATH, new
Path(appPath, "workflow.xml").toString)
workflowConfiguration.setProperty(OozieClient.USER_NAME, getTestUser)
usrDefinedProperties.foreach { case (k, v) =>
workflowConfiguration.put(k.toString, v.toString) }
val dagEngine:DagEngine =
Services.get.get(classOf[DagEngineService]).getDagEngine(getTestUser)
val conf = new Configuration()
workflowConfiguration.asScala.toMap.foreach{case(k,v ) => conf.set(k, v)}
val jobId = dagEngine.dryRunSubmit(conf)
But I would like to check how workflow flows when you pass in different
parameters without actualy executing steps.
