Hi Thomas, Sorry for my delayed response and thanks for your all of your work. Vagrant related
I see that the brooklyn-dist PR that you referenced [0] was indeed merged and I agree it appears that it would fix the symlink issue I’ve observed. However, when I download the Vagrant tar archive, I’m still getting the same MD5 sum that was produced back on September 27th: 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I’m using the command found at https://brooklyn.apache.org/#get-started : curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz " https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz " So, this yields a new question about the update process for mirrors. Is it working? Or perhaps I didn’t understand the release model with brooklyn-dist . Anyhow, I still need to apply the fix I mentioned previously (linking and restarting the service after vagrant up) [1]. Unfortunately, I’m still experiencing issues with the bento/centos-7.3 boxes too, so I’m continuing to change the box variable to geerlingguy/centos7. I appreciate your help in debugging this. You referenced an error message with the text “Could not find the X.Org or XFree86 Window System, skipping.” . Is this expected to work on macOS? Is there a simple method of testing the integration? My approach is still working for me, and once the service is running I can access Brooklyn from my host, at http://localhost:8081 . Deployment related After getting Brooklyn started, the thing I’m eager to use more than anything is Ambari. Here are the steps I’ve done, attempting deployment: # Install the Brooklyn command line tool brew install apache-brooklyn-cli # Authenticate to Brooklyn br login http://localhost:8081/ # Get the Brooklyn Ambari repo git clone https://github.com/brooklyncentral/brooklyn-ambari.git cd brooklyn-ambari # Add Ambari to Brooklyn's catalog br add-catalog catalog.bom Next, in AWS, I had to establish my Security Group. I first created a security group called test-ambari and opened ports 8080 according to the brooklyn-ambari README. This failed, reasonably, since I know Ambari needs 8440 for coordinating agent nodes. In a third or fourth iteration, I saw an error that referenced port 22. At that point, I decided to just open things up a lot wider, in hopes that the networking would get everything working properly. My final Security Group, with speculative rules for TCP connections from anywhere is the following: * 8080 # The Ambari web UI * 24007 # I saw mention of GlusterFS in the logs * 24008 # Again, for GlusterFS * 8441 # Ambari Registration and Heartbeat Port * 22 # It seems the Brooklyn control machine has to SSH to Ambari nodes * 8440 # Ambari Agent orchestration * 2181 # ZooKeeper Next, from the Brooklyn web UI, I navigate to the Composer / editor and enter this YAML: location: jclouds:aws-ec2: # edit these to use your credential (or delete if credentials specified in brooklyn.properties) identity: <redacted> credential: <redacted> region: us-east-2 # we want Ubuntu, with a lot of RAM osFamily: ubuntu minRam: 8gb # set up this user and password (default is to authorize a public key) user: sample password: s4mpl3 services: - type: ambari-cluster-application name: Ambari Cluster brooklyn.config: securityGroup: test-ambari initialSize: 3 services: - FALCON - FLUME - GANGLIA - HBASE - HDFS - KAFKA - KERBEROS - MAPREDUCE2 - OOZIE - PIG - SLIDER - SQOOP - YARN - ZOOKEEPER After clicking the Deploy button, 4 instances are created in AWS. Here’s a screenshot: https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view I can watch, in the Brooklyn UI, that there is communication between the Brooklyn Vagrant VM and these EC2 hosts. Screenshot: https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view Eventually, all 3 Ambari agent nodes seem to be in a happy state. Unfortunately, the Ambari Server is not: Screenshot: https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view Screenshot: https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view When I attempt to open port 8080 (Ambari web UI) on the Ambari server, I’m receiving an error. Screenshot: https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view I know that at one point, things were working to a greater degree than I am seeing now. Unfortunately, I don’t recall how I managed to accomplish that (maybe using the newer BOM file from this PR [2] or perhaps it was the BYON / Vagrant nodes). I was, at one point, able to login to Ambari. I found it really great that there was a Brooklyn generated password for the service. As a last ditch effort for today, I did try the former on AWS and didn’t have success. Wrapping up My team and I need move quickly, and unfortunately, if I can’t get things working with Brooklyn soon - I’ll need to change my approach. I think the Brooklyn team has a very serious opportunity if you can support Ambari. I say that, since I’m not totally satisfied with the job that Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem and Ambari. Presumably, if you can support the baseline, you’ll inherit a variety of other services. Furthermore, since Brooklyn has first-class support for load balancing and resizing, Hadoop users get serious value in being able to scale workloads. The possibility of developing / testing distributed workloads on local VMs is another value not so well supported in the open source. Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn configuration), then Brooklyn could provide a very rich feature set for securing clusters, data, and custom endpoints. Perhaps some other Apache folks would be willing to help with this integration? Thanks for all your help, b-long 0: https://github.com/apache/brooklyn-dist/pull/111 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126
