Hi Brian, I'm happy to help with the brooklyn-ambari issues you are having - and I'm don't have much in my calendar today so please email me directly if you like to screenshare using skype or google hangouts.
I think that you're correct that the issues you are seeing are down to your security group config. Depending on how you are using AWS there are a couple of intricacies with how you configure your security. But usually the important issue is that you open all ports for traffic within the security group. There's also the possibility that you location config is using iptables which, in general, needs to be switched off. I haven't deployed to AWS in a while so I'll go through the steps that you took and then I'll follow up this email with some screenshots and further troubleshooting steps. (It'll probably take me about an hour - a full installation is not the quickest thing in the world). Regards Duncan On Thu, 5 Oct 2017 at 21:00 Brian Long <[email protected]> wrote: > Hi all, > > Sorry for the spam -- I'm just getting used to the plain text mailing > list. If you prefer HTML formatting, I've uploaded my message here: > https://gist.github.com/b-long/9156a3696d9c30c14a092fe4b0a01381 > > Thanks, > b-long > > On Thu, Oct 5, 2017 at 3:12 PM, Brian Long <[email protected]> wrote: > > Hi Thomas, > > > > Sorry for my delayed response and thanks for your all of your work. > > > > Vagrant related > > > > I see that the brooklyn-dist PR that you referenced [0] was indeed merged > > and I agree it appears that it would fix the symlink issue I’ve observed. > > However, when I download the Vagrant tar archive, I’m still getting the > same > > MD5 sum that was produced back on September 27th: > > 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I’m > > using the command found at https://brooklyn.apache.org/#get-started : > > curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz > > " > https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz > " > > > > So, this yields a new question about the update process for mirrors. Is > it > > working? Or perhaps I didn’t understand the release model with > > brooklyn-dist. > > > > Anyhow, I still need to apply the fix I mentioned previously (linking and > > restarting the service after vagrant up) [1]. Unfortunately, I’m still > > experiencing issues with the bento/centos-7.3 boxes too, so I’m > continuing > > to change the box variable to geerlingguy/centos7. I appreciate your > help in > > debugging this. You referenced an error message with the text “Could not > > find the X.Org or XFree86 Window System, skipping.” . Is this expected to > > work on macOS? Is there a simple method of testing the integration? > > > > My approach is still working for me, and once the service is running I > can > > access Brooklyn from my host, at http://localhost:8081 . > > > > Deployment related > > > > After getting Brooklyn started, the thing I’m eager to use more than > > anything is Ambari. Here are the steps I’ve done, attempting deployment: > > > > # Install the Brooklyn command line tool > > brew install apache-brooklyn-cli > > > > # Authenticate to Brooklyn > > br login http://localhost:8081/ > > > > # Get the Brooklyn Ambari repo > > git clone https://github.com/brooklyncentral/brooklyn-ambari.git > > > > cd brooklyn-ambari > > > > # Add Ambari to Brooklyn's catalog > > br add-catalog catalog.bom > > > > Next, in AWS, I had to establish my Security Group. I first created a > > security group called test-ambari and opened ports 8080 according to the > > brooklyn-ambari README. This failed, reasonably, since I know Ambari > needs > > 8440 for coordinating agent nodes. In a third or fourth iteration, I saw > an > > error that referenced port 22. At that point, I decided to just open > things > > up a lot wider, in hopes that the networking would get everything working > > properly. My final Security Group, with speculative rules for TCP > > connections from anywhere is the following: > > > > * 8080 # The Ambari web UI > > * 24007 # I saw mention of GlusterFS in the logs > > * 24008 # Again, for GlusterFS > > * 8441 # Ambari Registration and Heartbeat Port > > * 22 # It seems the Brooklyn control machine has to SSH to Ambari > nodes > > * 8440 # Ambari Agent orchestration > > * 2181 # ZooKeeper > > > > Next, from the Brooklyn web UI, I navigate to the Composer / editor and > > enter this YAML: > > > > location: > > jclouds:aws-ec2: > > # edit these to use your credential (or delete if credentials > specified > > in brooklyn.properties) > > identity: <redacted> > > credential: <redacted> > > > > region: us-east-2 > > > > # we want Ubuntu, with a lot of RAM > > osFamily: ubuntu > > minRam: 8gb > > > > # set up this user and password (default is to authorize a public > key) > > user: sample > > password: s4mpl3 > > > > services: > > - type: ambari-cluster-application > > name: Ambari Cluster > > brooklyn.config: > > securityGroup: test-ambari > > initialSize: 3 > > services: > > - FALCON > > - FLUME > > - GANGLIA > > - HBASE > > - HDFS > > - KAFKA > > - KERBEROS > > - MAPREDUCE2 > > - OOZIE > > - PIG > > - SLIDER > > - SQOOP > > - YARN > > - ZOOKEEPER > > > > After clicking the Deploy button, 4 instances are created in AWS. Here’s > a > > screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view > > > > I can watch, in the Brooklyn UI, that there is communication between the > > Brooklyn Vagrant VM and these EC2 hosts. Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view > > > > Eventually, all 3 Ambari agent nodes seem to be in a happy state. > > Unfortunately, the Ambari Server is not: > > > > Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view > > > > Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view > > > > When I attempt to open port 8080 (Ambari web UI) on the Ambari server, > I’m > > receiving an error. Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view > > > > I know that at one point, things were working to a greater degree than I > am > > seeing now. Unfortunately, I don’t recall how I managed to accomplish > that > > (maybe using the newer BOM file from this PR [2] or perhaps it was the > BYON > > / Vagrant nodes). I was, at one point, able to login to Ambari. I found > it > > really great that there was a Brooklyn generated password for the > service. > > As a last ditch effort for today, I did try the former on AWS and didn’t > > have success. > > > > Wrapping up > > > > My team and I need move quickly, and unfortunately, if I can’t get things > > working with Brooklyn soon - I’ll need to change my approach. > > > > I think the Brooklyn team has a very serious opportunity if you can > support > > Ambari. I say that, since I’m not totally satisfied with the job that > > Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem > > and Ambari. Presumably, if you can support the baseline, you’ll inherit a > > variety of other services. > > > > Furthermore, since Brooklyn has first-class support for load balancing > and > > resizing, Hadoop users get serious value in being able to scale > workloads. > > The possibility of developing / testing distributed workloads on local > VMs > > is another value not so well supported in the open source. > > > > Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn > > configuration), then Brooklyn could provide a very rich feature set for > > securing clusters, data, and custom endpoints. Perhaps some other Apache > > folks would be willing to help with this integration? > > > > Thanks for all your help, > > b-long > > > > 0: https://github.com/apache/brooklyn-dist/pull/111 > > 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22 > > 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126 >
