Yes support for Spark was specifically added through https://issues.apache.org/jira/browse/OOZIE-1983, to allow users of Oozie or Falcon to run Spark jobs. Moving the retention job to Spark would create a heavy dependency on spark within Falcon.
With https://issues.apache.org/jira/browse/FALCON-965, it should be possible to create an alternate implementation of eviction. Regards Srikanth Sundarrajan > Date: Tue, 3 Feb 2015 15:51:38 +0100 > From: [email protected] > To: [email protected] > Subject: Falcon with Spark ? > > Hi all, > > I'm working (and finally resuming my work ;)) on some Falcon features: > - Update and improvements on the ActiveMQ broker > - Complete CDC support of diff/gap storage > - Support of more workflow entities (mapreduce directly instead of Oozie > workflow.xml, etc) > > For the workflow entities, I would like to evaluate the "direct" support > of Spark. > Generally speaking, I wonder if "oppositionally", we couldn't leverage > Spark for some internal Falcon processes (like eviction, etc). > > WDYT ? > > Regards > JB > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com
