Hi all, Sorry for the spam -- I'm just getting used to the plain text mailing list. If you prefer HTML formatting, I've uploaded my message here: https://gist.github.com/b-long/9156a3696d9c30c14a092fe4b0a01381
Thanks, b-long On Thu, Oct 5, 2017 at 3:12 PM, Brian Long <brianbl...@gmail.com> wrote: > Hi Thomas, > > Sorry for my delayed response and thanks for your all of your work. > > Vagrant related > > I see that the brooklyn-dist PR that you referenced [0] was indeed merged > and I agree it appears that it would fix the symlink issue I’ve observed. > However, when I download the Vagrant tar archive, I’m still getting the same > MD5 sum that was produced back on September 27th: > 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I’m > using the command found at https://brooklyn.apache.org/#get-started : > curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz > "https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz"; > > So, this yields a new question about the update process for mirrors. Is it > working? Or perhaps I didn’t understand the release model with > brooklyn-dist. > > Anyhow, I still need to apply the fix I mentioned previously (linking and > restarting the service after vagrant up) [1]. Unfortunately, I’m still > experiencing issues with the bento/centos-7.3 boxes too, so I’m continuing > to change the box variable to geerlingguy/centos7. I appreciate your help in > debugging this. You referenced an error message with the text “Could not > find the X.Org or XFree86 Window System, skipping.” . Is this expected to > work on macOS? Is there a simple method of testing the integration? > > My approach is still working for me, and once the service is running I can > access Brooklyn from my host, at http://localhost:8081 . > > Deployment related > > After getting Brooklyn started, the thing I’m eager to use more than > anything is Ambari. Here are the steps I’ve done, attempting deployment: > > # Install the Brooklyn command line tool > brew install apache-brooklyn-cli > > # Authenticate to Brooklyn > br login http://localhost:8081/ > > # Get the Brooklyn Ambari repo > git clone https://github.com/brooklyncentral/brooklyn-ambari.git > > cd brooklyn-ambari > > # Add Ambari to Brooklyn's catalog > br add-catalog catalog.bom > > Next, in AWS, I had to establish my Security Group. I first created a > security group called test-ambari and opened ports 8080 according to the > brooklyn-ambari README. This failed, reasonably, since I know Ambari needs > 8440 for coordinating agent nodes. In a third or fourth iteration, I saw an > error that referenced port 22. At that point, I decided to just open things > up a lot wider, in hopes that the networking would get everything working > properly. My final Security Group, with speculative rules for TCP > connections from anywhere is the following: > > * 8080 # The Ambari web UI > * 24007 # I saw mention of GlusterFS in the logs > * 24008 # Again, for GlusterFS > * 8441 # Ambari Registration and Heartbeat Port > * 22 # It seems the Brooklyn control machine has to SSH to Ambari nodes > * 8440 # Ambari Agent orchestration > * 2181 # ZooKeeper > > Next, from the Brooklyn web UI, I navigate to the Composer / editor and > enter this YAML: > > location: > jclouds:aws-ec2: > # edit these to use your credential (or delete if credentials specified > in brooklyn.properties) > identity: <redacted> > credential: <redacted> > > region: us-east-2 > > # we want Ubuntu, with a lot of RAM > osFamily: ubuntu > minRam: 8gb > > # set up this user and password (default is to authorize a public key) > user: sample > password: s4mpl3 > > services: > - type: ambari-cluster-application > name: Ambari Cluster > brooklyn.config: > securityGroup: test-ambari > initialSize: 3 > services: > - FALCON > - FLUME > - GANGLIA > - HBASE > - HDFS > - KAFKA > - KERBEROS > - MAPREDUCE2 > - OOZIE > - PIG > - SLIDER > - SQOOP > - YARN > - ZOOKEEPER > > After clicking the Deploy button, 4 instances are created in AWS. Here’s a > screenshot: > https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view > > I can watch, in the Brooklyn UI, that there is communication between the > Brooklyn Vagrant VM and these EC2 hosts. Screenshot: > https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view > > Eventually, all 3 Ambari agent nodes seem to be in a happy state. > Unfortunately, the Ambari Server is not: > > Screenshot: > https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view > > Screenshot: > https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view > > When I attempt to open port 8080 (Ambari web UI) on the Ambari server, I’m > receiving an error. Screenshot: > https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view > > I know that at one point, things were working to a greater degree than I am > seeing now. Unfortunately, I don’t recall how I managed to accomplish that > (maybe using the newer BOM file from this PR [2] or perhaps it was the BYON > / Vagrant nodes). I was, at one point, able to login to Ambari. I found it > really great that there was a Brooklyn generated password for the service. > As a last ditch effort for today, I did try the former on AWS and didn’t > have success. > > Wrapping up > > My team and I need move quickly, and unfortunately, if I can’t get things > working with Brooklyn soon - I’ll need to change my approach. > > I think the Brooklyn team has a very serious opportunity if you can support > Ambari. I say that, since I’m not totally satisfied with the job that > Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem > and Ambari. Presumably, if you can support the baseline, you’ll inherit a > variety of other services. > > Furthermore, since Brooklyn has first-class support for load balancing and > resizing, Hadoop users get serious value in being able to scale workloads. > The possibility of developing / testing distributed workloads on local VMs > is another value not so well supported in the open source. > > Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn > configuration), then Brooklyn could provide a very rich feature set for > securing clusters, data, and custom endpoints. Perhaps some other Apache > folks would be willing to help with this integration? > > Thanks for all your help, > b-long > > 0: https://github.com/apache/brooklyn-dist/pull/111 > 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22 > 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126
