Guijarro, Julio wrote: > Hi Stanley, > > > > See my answers below > > > > *From:* Stanley C. Burnitt [mailto:[email protected]]
> >3. Use smartfrog to deploy and run bash scripts which install > >cloudera.repo files, then install cloudera's hadoop / hbase / zookeeper > >distributions (via yum), edit configuration files, and issue startup & > >shutdown commands. > > Steve has been working on some especial components to deploy and manage > Hadoop but he is using Apache’s distro. He is on holiday for two weeks > and could give you an update on it once he is back. Hell, I'm back. I'm using my own patched version of trunk which -because it is trunk- is not something I'd keep files I value on. It's OK for short lived stuff, though as I haven't resynced with trunk for a while I will need to suffer to sync everything up and make sure that the reasons tests fail are not my fault. It's patched because I've given the hadoop services a common base class, something I'd like to get back in there but which is tricky to do as trying to keep up with Hadoop changes is a full time job, and I'm only doing the hadoop work in the spare time. With the base classes we can then then bring up Hadoop in the SF JVM, providing live configuration, monitoring liveness, etc. This is nice, but introduces surprises. we package up Hadoop in our own RPMs, just take the JARs I build myself and put them into the sf-hadoop RPM. This code only works with my modified hadoop version. >> 4. Use smartfrog to deploy and configure a solr cluster. (There is no >> yum pkg for solr, to my knowledge. Can I use smartfrog to deploy & run >> a bash script which scp's, extracts, and configures gzipped solr >> distributions on a cluster of solr servers?) > > There are components for SSH and SCP, you can also write components that > get the file from a webserver for example and perform the operation > locally. For example, you deploy a component in a machine that then > downloads a package locally, unzips it and run the app. I would advise > you to try the Dynamic Webserver example included with SF. It does > exactly what you describe here and you could reuse most of it. > You could also talk to the solr people about helping them create RPMs which suit your needs. > > 5. > Deploy and manage the dependencies and lifecycles of all of my > > system's OSGi components -- standalone product launchers on top of > > dozens of plugins. > > Does this seem reasonable? My entire system is implemented in java, and > for this reason I chose to look at smartfrog first, before bcfg2 or > puppet. But this is an OSGi based system, not J2EE, and I will be > deploying (then extracting) gzipped eclipse products, not jars. > Also, the OSGi bundles are started from native product launchers -- > again, not java jars. Julio, do we have any components to unzip stuff? The code is all in Java, there's also the ant tasks to do better things -but nothing in Java sets up filesystem permissions very well, as Sun didn't want you to be able to do that. If you want to unzip/untar with permissions, exec the native programs. > > With the stable release of SF you have to manage your OSGi components > using either Java and Scripts but there is a version of SF that has full > integration with OSGi (SF uses it but it also allows you to describe an > manage the entire environment but is not stable and has not yet been > used in production) The intention is to merge the branch into the trunk > soon but we do not know how long it is going to take to merge the code > and test it properly. > I'm looking forward to this too, but am sure it will make hadoop-under-SF even more fun. Know that following the merge we also plan to flip the license from LGPL to apache, just because people worry about the implications of LGPL and subclassing. >> And I intend to run system / scalability tests on EC2. We have some SF components to work with EC2 and S3, push out data and bring up/tear down VMs. There is a web front end, CloudFarmer, which does more: pushes out SmartFrog configurations to machines when you create them, works to bring up Hadoop clusters, etc. http://www.slideshare.net/steve_l/new-roles-for-the-cloud http://www.slideshare.net/steve_l/deploying-on-ec2 CloudFarmer has four back ends: mock, physical, EC2 and HP Cells, the latter two being API-driven. But it isn't HA, can leak VMs, which is not what you want when paying for VMs; you really need a recovery mechanism (either store metadata in the VM config field of each EC2 VM or use SimpleDB). And it needs security, though that's a matter of bringing up Jetty with the right certificate settings, which is a matter of configuring the SF jetty components right. If you want the HA features, you get to start coding. -Steve ------------------------------------------------------------------------------ The Palm PDK Hot Apps Program offers developers who use the Plug-In Development Kit to bring their C/C++ apps to Palm for a share of $1 Million in cash or HP Products. Visit us here for more details: http://ad.doubleclick.net/clk;226879339;13503038;l? http://clk.atdmt.com/CRS/go/247765532/direct/01/ _______________________________________________ Smartfrog-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/smartfrog-users
