Vote for approach 2. Apart from listed pros, Having recipe cooking on server side will make it easier for falcon to create a recipe repository in the future. Once we implement tag management in Falcon, Monitoring of recipe process status will become easier as well.
-1 for using Oozie. Thanks Balu On 3/12/15, 7:47 PM, "Venkat Ranganathan" <[email protected]> wrote: >It looks like the initial thought was to have the recipe processing a >client side feature because of isolation issues. It has its merits >(primarily in terms of allowing Falcon server insulated from the recipe >processing), client side recipes make the ability to catalog, query, >reuse >and management of the recipes a little difficult. > >I agree that we don¹t want JSPs. I don¹t think it is a good idea to >expect Java applet capability in the UI. It brings its own list of >security concerns. > >Using oozie seems like a natural approach given Falcon uses Oozie a lot. >But the operational issue with this approach is said to be the overall >apparatus needed to monitor and manage the recipe-cooking worfklows. >Oozie scheduling can bring arbitrary delays (as we have seen internally >also). Granted we can design around the limitations and make use of the >strengths of the approach but it seems something we can avoid if we can. > >Option 2 is to fork a new process in falcon server for Option 2 and may >be >we throttle how many processes we spawn at a time. We may have some >work in secure clusters, but this will give the necessary control over >the >recipe cooking while not needing additional operational work. We will >write this in such a way that if we choose to use the oozie apprach, the >code can be made as part of a Oozie java action with some minimal effort. > > >Thanks > >Venkat > >On 3/12/15, 12:53 PM, "Arpit Gupta" <[email protected]> wrote: > >>I fail to understand why is it best to keep recipe client side. >> >>Also if the user is allowed to run any arbitrary java code for recipe >>cooking and it runs in the browser how would we handle if the user code >>wanted to check something on hdfs? If the cluster was secure? Nothing >>over rpc would work. Then that means they cant write any arbitrary code. >>May be i am missing something in what kind of user code are we expecting. >> >>for option #2 will falcon handle forking off another process? >>-- >>Arpit Gupta >>Hortonworks Inc. >>http://hortonworks.com/ >> >>> On Mar 12, 2015, at 11:46 AM, Ying Zheng <[email protected]> >>>wrote: >>> >>> The first question is whether we need to move recipe to server side or >>> not, no matter which webpage we use (JSP or HTML5). As far as I >>> understand, it is best to keep it on client side. Yes? If we choose >>>HTML5, >>> one way is to use JAVA applet in HTML5. >>> >>> >>> On 3/12/15, 9:31 AM, "Siva Thumma" <[email protected]> wrote: >>> >>>> +1. >>>> >>>>> On 12-Mar-2015, at 8:58 pm, Arpit Gupta <[email protected]> >>>>>wrote: >>>>> >>>>> Are people not moving away from JSP's? I know all UI's in Hadoop >>>>>moved >>>>> away from JSP recently to html 5 >>>>> https://issues.apache.org/jira/browse/HADOOP-10563 >>>>> >>>>> -- >>>>> Arpit Gupta >>>>> Hortonworks Inc. >>>>> http://hortonworks.com/ >>>>> >>>>> On Mar 12, 2015, at 8:13 AM, Ying Zheng >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> >>>>> Sort of, but not a completely new server. We don't need to write a >>>>>new >>>>> server. As long as we have Tomcat installed and running, we just >>>>>need >>>>>to >>>>> copy front end JSP files to a folder under tomcat. In this way, we >>>>> separate front end and back end. >>>>> >>>>> On Mar 12, 2015 7:39 AM, Srikanth Sundarrajan >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> Are you suggesting a completely new server for cooking recipe? >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 12-Mar-2015, at 8:03 pm, "Ying Zheng" >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> >>>>> Perhaps I don't understand your question. JSP is one way to generate >>>>> webpages. Why does it need to run in Falcon server? It can be >>>>>isolated >>>>> from Falcon server, and developed and run in Apache Tomcat server. >>>>> >>>>> Thanks, >>>>> Ying >>>>> >>>>> On Mar 12, 2015 12:53 AM, Srikanth Sundarrajan >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> Hi Ying, >>>>> JSP would still run in Falcon server JVM, so not sure if that is any >>>>> different from implementing this natively inside the falcon server. >>>>> >>>>> Regards >>>>> Srikanth Sundarrajan >>>>> >>>>> ---------------------------------------- >>>>> Subject: Re: [DISCUSS]: Moving recipe cooking to server >>>>> From: [email protected]<mailto:[email protected]> >>>>> To: [email protected]<mailto:[email protected]> >>>>> Date: Thu, 12 Mar 2015 07:43:28 +0000 >>>>> >>>>> Hi Sowmya, Srikanth, Venkatesh and Venkat, >>>>> >>>>> As far as I understand, recipe helps the user generate the process >>>>> entity >>>>> xml and it looks best to keep it a client side logic, as what you did >>>>> before. The current question is how to run recipe logic on DR UI. To >>>>> avoid >>>>> repeating all the recipe logic in JS, one choice you can consider is >>>>> JSP. >>>>> JSP can be supported by Apache Tomcat server. If you package the >>>>>recipe >>>>> code into JAR, it can be reused in JSP. JS and CSS can also be >>>>>included >>>>> in >>>>> the JSP page. >>>>> >>>>> Cheers, >>>>> Ying >>>>> >>>>> >>>>> On 3/11/15, 11:51 PM, "Srikanth Sundarrajan" >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> >>>>> Thanks Sowmya for capturing the discussion and also initiating this >>>>> conversation to drive consensus around the design. I am inclined >>>>>towards >>>>> Approach 2. >>>>> >>>>> Regards >>>>> Srikanth Sundarrajan >>>>> >>>>> ---------------------------------------- >>>>> Subject: [DISCUSS]: Moving recipe cooking to server >>>>> From: [email protected]<mailto:[email protected]> >>>>> To: [email protected]<mailto:[email protected]> >>>>> Date: Thu, 12 Mar 2015 05:17:16 +0000 >>>>> >>>>> I had a discussion with Srikanth, Venkatesh and Venkat regarding >>>>>making >>>>> recipe processing server side concept. I am sending out the summary >>>>>of >>>>> the meeting and opening it for further discussion. >>>>> >>>>> Today Recipe cooking is a client side logic. Recipe also supports >>>>> extensions i.e. user can cook his/her own custom recipes. >>>>> Decision to make it client side logic was for the following reasons >>>>> >>>>> * Keep it isolated from falcon server >>>>> >>>>> * As custom recipe cooking is supported, user recipes can introduce >>>>> security vulnerabilities and also can bring down the falcon server >>>>> >>>>> Today, falcon provides HDFS DR recipe out of the box. There is a plan >>>>> to add UI support for DR in Falcon. >>>>> Rest API support cannot be added for recipe as it is client side >>>>> processing. >>>>> If the UI is pure java script[JS] then all the recipe cooking logic >>>>>has >>>>> to be repeated in JS. This is not a feasible solution - if more >>>>>recipes >>>>> are added say DR for hive, hbase and others, UI won't be extensible. >>>>> >>>>> For the above mentioned reasons Recipe should me made a server side >>>>> logic. >>>>> Provided recipes [recipes provided out of the box] can run as Falcon >>>>> process. Recipe cooking will be done in a new process if its custom >>>>> recipe [user code]. >>>>> >>>>> For cooking of custom recipes, design proposed should consider >>>>>handling >>>>> security implications, handling the issues where the custom user code >>>>> can bring down the Falcon server (trapping System.exit), handling >>>>>class >>>>> path isolation. >>>>> Also it shouldn't in anyway destabilize the Falcon system. >>>>> >>>>> There are couple of approaches which was discussed >>>>> >>>>> Approach 1: >>>>> Custom Recipe cooking can be carried out separately in another Oozie >>>>> WF, this will ensure isolation. Oozie already has the ability to >>>>> schedule jobs as a user and handles all the security aspects of it. >>>>> >>>>> Pros: >>>>> - Provides isolation >>>>> - Piggyback on Oozie as it already provides the required >>>>>functionality >>>>> >>>>> Cons: >>>>> - As recipe processing is done in different WF, from operations point >>>>> of view user cannot figure out recipe processing status and thus >>>>>adds >>>>>to >>>>> the operational pain. >>>>> >>>>> Approach 2: >>>>> Custom recipe cooking is done on the server side in a separate >>>>> independent process than Falcon process I.e. It runs in a different >>>>>JVM. >>>>> Throttling should be added for how many recipe cooking processes can >>>>>be >>>>> launched keeping in mind the machine configuration. >>>>> >>>>> Pros: >>>>> - Provides isolation as recipe cooking is done in a independent >>>>>process >>>>> >>>>> Cons: >>>>> - Performance overhead as new process is launched for custom recipe >>>>> cooking >>>>> - Adds more complexity to the system >>>>> >>>>> Thoughts? >>>>> >>>>> Thanks, >>>>> Sowmya >>>>> >>>>> >>> >>
