There are a couple of questions before I create a jira. Is this a client-side only concept? Do we provide a server-side recipe tool as well so web apps can use this instead. The downside to using the client side api is that it expects kinit to have been done.
How do we handle packaging? There could be recipe dependent jars, libs and this cannot be released in apache. Thoughts? On Fri, Aug 8, 2014 at 2:46 PM, Seetharam Venkatesh <[email protected] > wrote: > > > > On Thu, Aug 7, 2014 at 10:25 PM, Srikanth Sundarrajan <[email protected] > > wrote: > >> +1 for the capability. >> >> Some clarifications. >> >> 1. Is creation of recipe in user space or would that need to be bundled >> in falcon as part of distribution ? > > The thinking is to have it bundled in falcon as part of the distribution > to start with. > > User space is quite nice but falcon recipe can look at the templates dir > to find one and then substitute the properties. This might be possible too. > > >> How do users provide support artifacts for the recipe >> > All artifacts must be on HDFS. > > >> 2. How do users track execution of instances of a particular recipe. Am >> assuming you can create multiple instances of a single recipe >> > Yes, name can also be templated and parameterized. > > >> 3. Would simple properties table suffice? >> > Thats the thinking. Sowmya has prototyped DR running on a source cluster. > > >> >> Regards >> Srikanth Sundarrajan >> >> > Date: Thu, 7 Aug 2014 19:34:13 -0700 >> > Subject: [DISCUSS] Recipes in Falcon >> > From: [email protected] >> > To: [email protected] >> > >> > Hi folks, >> > >> > Falcon offers many services OOTB and caters to a wide array of use >> cases. >> > However, there has been many asks that does not fit the functionality >> > offered by Falcon. I'm proposing that we add recipes to Falcon which is >> > similar to recipes in Whirr and other management solutions such as >> puppet >> > and chef. >> > >> > Overview: >> > A recipe essentially is a static process template with parameterized >> > workflow to realize a specific use case. For example: >> > >> > * replicating directories from one HDFS cluster to another (not timed >> > partitions) >> > * replicating hive metadata (database, table, views, etc.) >> > * replicating between HDFS and Hive - either way >> > * anonymization of data based on schema >> > * data masking >> > * etc. >> > >> > Proposal: >> > Falcon provides a Process abstraction that encapsulates the >> configuration >> > for a user workflow with scheduling controls. All recipes can be modeled >> > as a Process with in Falcon which executes the user workflow >> > periodically. The process and its associated workflow are parameterized. >> > The user will provide a properties file with name value pairs that are >> > substituted by falcon before scheduling it. >> > >> > The CLI would look something like this: >> > falcon -recipe $recipe_name -properties $properties_file >> > >> > Thoughts? >> > >> > -- >> > Regards, >> > Venkatesh >> > >> > “Perfection (in design) is achieved not when there is nothing more to >> add, >> > but rather when there is nothing more to take away.” >> > - Antoine de Saint-Exupéry >> >> > > > > -- > Regards, > Venkatesh > > “Perfection (in design) is achieved not when there is nothing more to add, > but rather when there is nothing more to take away.” > - Antoine de Saint-Exupéry > -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
