In regards to #6, I'm curious if you've considered using Yammer's Apache licensed metrics library [1] for capturing stats as it offers many different options for consuming and monitoring produced metrics. Coda Hale gives a good overview in this talk [2].
[1]: http://metrics.codahale.com/ [2]: http://pivotallabs.com/139-metrics-metrics-everywhere/ - Dave On Apr 13, 2013, at 12:15 PM, Andres Danter <[email protected]> wrote: > Awesome suggestions! Thank you, Josh. > On Apr 12, 2013 10:01 PM, "Josh Elser" <[email protected]> wrote: > >> Andres, >> >> Looks good. Glad to see a lot of good ideas here already. I have a couple >> of refinements if you don't mind. >> >> 3. Have the ability to schedule /graceful/ restart of Accumulo... >> >> It would be really cool to have the option of bringing down an entire >> cluster or do a rolling-restart (iteratively bring down a node, let things >> quiesce, then restart that node) to try to ensure 100% availability. >> Accumulo handles most of this for you -- it's more a matter of waiting for >> any unhosted tablets to be re-assigned after a tabletserver is stopped. >> >> 4. Push Accumulo code ... >> >> With 1.5 releases, there's a VFS classloader built in which allows for >> various types of file systems to be used to load files onto the classpath. >> The most interesting of which probably include HDFS, HTTP(S), and (S)FTP. >> It would be neat to support not only loading classes from the local >> filesystem but also by a variety of other means. >> >> 6. Gather system and/or application metrics.. >> >> Just be aware that this is a very open-ended subject and, most likely, >> could be a full GSOC task on its own. You probably want to think about what >> might be best for Accumulo (starting with what the monitor and master >> already supply) and how best you can integrate with Ambari. >> >> Again, overall, sounds awesome. I look forward to working with you. >> >> >> On 04/12/2013 05:58 PM, Andres Danter wrote: >> >>> Thanks, Billie. I'm liking this task more and more. I would love to take >>> this on for GSoC. Please run it by the Ambari PMC. I will start drafting >>> the proposal this weekend. >>> >>> Just to be clear, the goal would be to integrate Ambari with Accumulo in >>> order to achieve the following capabilities (I've expanded David's >>> original >>> list of features): >>> >>> 1. Control (start/stop) Accumulo processes on any cluster node >>> 2. Have the ability to edit Accumulo configuration files via a web-based >>> editor (I don't think Ambari provides this. Please let me know if I'm >>> wrong) >>> 3. Have the ability to schedule restart of Accumulo processes after a >>> configuration change >>> 4. Push Accumulo code and configuration files to a new cluster node >>> 5. Gracefully remove a node from a cluster >>> 6. Gather system and/or application metrics that can be used to monitor >>> the system and perhaps create reports on usability >>> 7. Be able to monitor and control the Accumulo cluster from a web-based >>> GUI (this kind of goes with #2 above) >>> 8. Allow read-only monitor sessions, as well as full-admin sessions and >>> have the ability to strictly manage who can access either. (this I got >>> from ACCUMULO-196) >>> 9. Provide the ability to authenticate user sessions with SSL >>> (see ACCUMULO-196) >>> >>> It seems like the product of this work would largely replace the current >>> Accumulo Monitor. If that is the case, are there features of the monitor >>> which you would like to see integrated? >>> >>> I will have about 3 months to complete this work (actually about 220-240 >>> hours), so that list is doable. Most of the work will involve the GUI >>> portions, unless Ambari already comes with much of the functionality. If >>> the list is too short, please let me know. >>> >>> I guess I should sign up to the Ambari mailing list as well, right? >>> >>> Thanks again and I look forward to being your mentee and to working with >>> the Accumulo community. >>> >>> Andres >>> >>> >>> On Fri, Apr 12, 2013 at 3:43 PM, David Medinets <[email protected] >>>> **wrote: >>> >>> The project is open source ... we can't stop you! >>>> >>>> >>>> On Fri, Apr 12, 2013 at 2:25 PM, Andres Danter <[email protected]> >>>> wrote: >>>> >>>> Could the work of integrating Accumulo with Ambari be something that you >>>>> would allow a student to tackle for GSoC? I don't want to step on >>>> anyone's >>>> >>>>> planned work, but such a task really interests me. I want to tackle >>>>> something with some meat, if you know what I mean. >>>>> >>>>> >>>>> >>>>> On Fri, Apr 12, 2013 at 11:47 AM, David Medinets >>>>> <[email protected]>**wrote: >>>>> >>>>> Thanks for the reminder. I had forgotten the name of Ambari. >>>>>> >>>>>> >>>>>> On Fri, Apr 12, 2013 at 10:37 AM, Keith Turner <[email protected]> >>>>> wrote: >>>> >>>>> On Fri, Apr 12, 2013 at 10:29 AM, David Medinets >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> It might be interesting to have a tool which could: >>>>>>>> >>>>>>>> a) shut down any accumulo process on any server. >>>>>>>> b) start any accumulo process on any server. >>>>>>>> c) edit the various configuration files through a web interface. >>>>>>>> d) automate restart accumulo processes, as needed, after >>>>>>> configuration >>>>> >>>>>> change. >>>>>>>> e) push accumulo code and configuration to a new server. >>>>>>>> f) gracefully shutdown and remove a server from the cluster. >>>>>>>> >>>>>>>> Anything else? If this sounds like a good idea, let's create a JIRA >>>>>>> ticket >>>>>>> >>>>>>>> with the right tags. >>>>>>> This sounds like Apache Ambari, which we should possibly integrate >>>>>>> with. See ACCUMULO-136 >>
