Created https://issues.apache.org/jira/browse/FALCON-634
On Tue, Aug 19, 2014 at 10:18 PM, <[email protected]> wrote: > Hi guys, > > It sounds good. I'm back after a "long" period on customers engagements ;) > > I think server-side for recipe is interesting as it would allow ready to > use webapp stuff. > > Regarding the packaging, it could be part of falcon-extra: in other Apache > projects (like Camel, ServiceMix, etc), when I have a non Apache compliant > "extension" (for instance in terms of license), I put it in *-extra. > It's hosted on github, approved by Apache but not under the Apache "scope" > (about release process vote, license, etc). > > Regards > JB > > > On 2014-08-20 00:17, Seetharam Venkatesh wrote: > >> There are a couple of questions before I create a jira. >> >> Is this a client-side only concept? >> Do we provide a server-side recipe tool as well so web apps can use this >> instead. The downside to using the client side api is that it expects >> kinit >> to have been done. >> >> How do we handle packaging? >> There could be recipe dependent jars, libs and this cannot be released in >> apache. >> >> Thoughts? >> >> >> >> On Fri, Aug 8, 2014 at 2:46 PM, Seetharam Venkatesh < >> [email protected] >> >>> wrote: >>> >> >> >>> >>> >>> On Thu, Aug 7, 2014 at 10:25 PM, Srikanth Sundarrajan < >>> [email protected] >>> > wrote: >>> >>> +1 for the capability. >>>> >>>> Some clarifications. >>>> >>>> 1. Is creation of recipe in user space or would that need to be bundled >>>> in falcon as part of distribution ? >>>> >>> >>> The thinking is to have it bundled in falcon as part of the distribution >>> to start with. >>> >>> User space is quite nice but falcon recipe can look at the templates dir >>> to find one and then substitute the properties. This might be possible >>> too. >>> >>> >>> How do users provide support artifacts for the recipe >>>> >>>> All artifacts must be on HDFS. >>> >>> >>> 2. How do users track execution of instances of a particular recipe. Am >>>> assuming you can create multiple instances of a single recipe >>>> >>>> Yes, name can also be templated and parameterized. >>> >>> >>> 3. Would simple properties table suffice? >>>> >>>> Thats the thinking. Sowmya has prototyped DR running on a source >>> cluster. >>> >>> >>> >>>> Regards >>>> Srikanth Sundarrajan >>>> >>>> > Date: Thu, 7 Aug 2014 19:34:13 -0700 >>>> > Subject: [DISCUSS] Recipes in Falcon >>>> > From: [email protected] >>>> > To: [email protected] >>>> > >>>> > Hi folks, >>>> > >>>> > Falcon offers many services OOTB and caters to a wide array of use >>>> cases. >>>> > However, there has been many asks that does not fit the functionality >>>> > offered by Falcon. I'm proposing that we add recipes to Falcon which >>>> is >>>> > similar to recipes in Whirr and other management solutions such as >>>> puppet >>>> > and chef. >>>> > >>>> > Overview: >>>> > A recipe essentially is a static process template with parameterized >>>> > workflow to realize a specific use case. For example: >>>> > >>>> > * replicating directories from one HDFS cluster to another (not timed >>>> > partitions) >>>> > * replicating hive metadata (database, table, views, etc.) >>>> > * replicating between HDFS and Hive - either way >>>> > * anonymization of data based on schema >>>> > * data masking >>>> > * etc. >>>> > >>>> > Proposal: >>>> > Falcon provides a Process abstraction that encapsulates the >>>> configuration >>>> > for a user workflow with scheduling controls. All recipes can be >>>> modeled >>>> > as a Process with in Falcon which executes the user workflow >>>> > periodically. The process and its associated workflow are >>>> parameterized. >>>> > The user will provide a properties file with name value pairs that are >>>> > substituted by falcon before scheduling it. >>>> > >>>> > The CLI would look something like this: >>>> > falcon -recipe $recipe_name -properties $properties_file >>>> > >>>> > Thoughts? >>>> > >>>> > -- >>>> > Regards, >>>> > Venkatesh >>>> > >>>> > “Perfection (in design) is achieved not when there is nothing more to >>>> add, >>>> > but rather when there is nothing more to take away.” >>>> > - Antoine de Saint-Exupéry >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Venkatesh >>> >>> “Perfection (in design) is achieved not when there is nothing more to >>> add, >>> but rather when there is nothing more to take away.” >>> - Antoine de Saint-Exupéry >>> >>> -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
