Hi Brian. Thanks for the very detailed email/gist, this kind of feedback is very valuable to the community and help us deliver the best possible experience.
Regarding your vagrant issues: - downloading the released vagrant version 0.12.0[1] won't work unfortunately. This was released as part of Brooklyn 0.12.0 and contains the bug you mentioned on your first email. We don't re-release under the same version therefore what you need to download is the next SNAPSHOT version available here[2]. This contains the fix I made for the symlink (my apologies, I should have mentioned it to you earlier) - the `bento/centos-7.3` box does work on macOS (I am running macOS myself) We choose this box over the official one for several reasons[3]. The error message I mentioned was present in the logs you provided in your first email. Not sure exactly what is going on here but I cannot reproduce this on my environment unfortunately. Anyhow, swapping you box for `geerlingguy/centos7` is fine as long as it works for you: that's the most important thing! Regarding your brooklyn-ambari issues, I can help you on this. Although, I see that Duncan already jumped in so it is up to you. Best. [1] https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz [2] https://repository.apache.org/service/local/artifact/maven/redirect?r=snapshots&g=org.apache.brooklyn&a=brooklyn-vagrant&v=0.13.0-SNAPSHOT&c=dist&e=tar.gz [3] https://github.com/apache/brooklyn-dist/pull/86 On Thu, 5 Oct 2017 at 20:00 Brian Long <[email protected]> wrote: > Hi all, > > Sorry for the spam -- I'm just getting used to the plain text mailing > list. If you prefer HTML formatting, I've uploaded my message here: > https://gist.github.com/b-long/9156a3696d9c30c14a092fe4b0a01381 > > Thanks, > b-long > > On Thu, Oct 5, 2017 at 3:12 PM, Brian Long <[email protected]> wrote: > > Hi Thomas, > > > > Sorry for my delayed response and thanks for your all of your work. > > > > Vagrant related > > > > I see that the brooklyn-dist PR that you referenced [0] was indeed merged > > and I agree it appears that it would fix the symlink issue I’ve observed. > > However, when I download the Vagrant tar archive, I’m still getting the > same > > MD5 sum that was produced back on September 27th: > > 331ab054e08a0b8c0480621b2f2adfe4 . To download the Vagrant archive, I’m > > using the command found at https://brooklyn.apache.org/#get-started : > > curl -SL --output apache-brooklyn-0.12.0-vagrant.tar.gz > > " > https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.12.0/apache-brooklyn-0.12.0-vagrant.tar.gz > " > > > > So, this yields a new question about the update process for mirrors. Is > it > > working? Or perhaps I didn’t understand the release model with > > brooklyn-dist. > > > > Anyhow, I still need to apply the fix I mentioned previously (linking and > > restarting the service after vagrant up) [1]. Unfortunately, I’m still > > experiencing issues with the bento/centos-7.3 boxes too, so I’m > continuing > > to change the box variable to geerlingguy/centos7. I appreciate your > help in > > debugging this. You referenced an error message with the text “Could not > > find the X.Org or XFree86 Window System, skipping.” . Is this expected to > > work on macOS? Is there a simple method of testing the integration? > > > > My approach is still working for me, and once the service is running I > can > > access Brooklyn from my host, at http://localhost:8081 . > > > > Deployment related > > > > After getting Brooklyn started, the thing I’m eager to use more than > > anything is Ambari. Here are the steps I’ve done, attempting deployment: > > > > # Install the Brooklyn command line tool > > brew install apache-brooklyn-cli > > > > # Authenticate to Brooklyn > > br login http://localhost:8081/ > > > > # Get the Brooklyn Ambari repo > > git clone https://github.com/brooklyncentral/brooklyn-ambari.git > > > > cd brooklyn-ambari > > > > # Add Ambari to Brooklyn's catalog > > br add-catalog catalog.bom > > > > Next, in AWS, I had to establish my Security Group. I first created a > > security group called test-ambari and opened ports 8080 according to the > > brooklyn-ambari README. This failed, reasonably, since I know Ambari > needs > > 8440 for coordinating agent nodes. In a third or fourth iteration, I saw > an > > error that referenced port 22. At that point, I decided to just open > things > > up a lot wider, in hopes that the networking would get everything working > > properly. My final Security Group, with speculative rules for TCP > > connections from anywhere is the following: > > > > * 8080 # The Ambari web UI > > * 24007 # I saw mention of GlusterFS in the logs > > * 24008 # Again, for GlusterFS > > * 8441 # Ambari Registration and Heartbeat Port > > * 22 # It seems the Brooklyn control machine has to SSH to Ambari > nodes > > * 8440 # Ambari Agent orchestration > > * 2181 # ZooKeeper > > > > Next, from the Brooklyn web UI, I navigate to the Composer / editor and > > enter this YAML: > > > > location: > > jclouds:aws-ec2: > > # edit these to use your credential (or delete if credentials > specified > > in brooklyn.properties) > > identity: <redacted> > > credential: <redacted> > > > > region: us-east-2 > > > > # we want Ubuntu, with a lot of RAM > > osFamily: ubuntu > > minRam: 8gb > > > > # set up this user and password (default is to authorize a public > key) > > user: sample > > password: s4mpl3 > > > > services: > > - type: ambari-cluster-application > > name: Ambari Cluster > > brooklyn.config: > > securityGroup: test-ambari > > initialSize: 3 > > services: > > - FALCON > > - FLUME > > - GANGLIA > > - HBASE > > - HDFS > > - KAFKA > > - KERBEROS > > - MAPREDUCE2 > > - OOZIE > > - PIG > > - SLIDER > > - SQOOP > > - YARN > > - ZOOKEEPER > > > > After clicking the Deploy button, 4 instances are created in AWS. Here’s > a > > screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzLTRHcTdTZFkzWG8/view > > > > I can watch, in the Brooklyn UI, that there is communication between the > > Brooklyn Vagrant VM and these EC2 hosts. Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzR0R2bFFDYWhCUk0/view > > > > Eventually, all 3 Ambari agent nodes seem to be in a happy state. > > Unfortunately, the Ambari Server is not: > > > > Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzZzZ1XzF1UmZQZkU/view > > > > Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzYkQ4N1dXeVVnUEU/view > > > > When I attempt to open port 8080 (Ambari web UI) on the Ambari server, > I’m > > receiving an error. Screenshot: > > https://drive.google.com/file/d/0B91hmcyKP8SzUkF2MGJWeFNCNDQ/view > > > > I know that at one point, things were working to a greater degree than I > am > > seeing now. Unfortunately, I don’t recall how I managed to accomplish > that > > (maybe using the newer BOM file from this PR [2] or perhaps it was the > BYON > > / Vagrant nodes). I was, at one point, able to login to Ambari. I found > it > > really great that there was a Brooklyn generated password for the > service. > > As a last ditch effort for today, I did try the former on AWS and didn’t > > have success. > > > > Wrapping up > > > > My team and I need move quickly, and unfortunately, if I can’t get things > > working with Brooklyn soon - I’ll need to change my approach. > > > > I think the Brooklyn team has a very serious opportunity if you can > support > > Ambari. I say that, since I’m not totally satisfied with the job that > > Hortonworks is doing supporting FLOSS deployments of the Hadoop ecosystem > > and Ambari. Presumably, if you can support the baseline, you’ll inherit a > > variety of other services. > > > > Furthermore, since Brooklyn has first-class support for load balancing > and > > resizing, Hadoop users get serious value in being able to scale > workloads. > > The possibility of developing / testing distributed workloads on local > VMs > > is another value not so well supported in the open source. > > > > Lastly, if we could get Apache Ranger working (via an Ambari + Brooklyn > > configuration), then Brooklyn could provide a very rich feature set for > > securing clusters, data, and custom endpoints. Perhaps some other Apache > > folks would be willing to help with this integration? > > > > Thanks for all your help, > > b-long > > > > 0: https://github.com/apache/brooklyn-dist/pull/111 > > 1: https://gist.github.com/b-long/ab096f45a7867574b74f01adff9f6c22 > > 2: https://github.com/brooklyncentral/brooklyn-ambari/pull/126 > -- Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation • https://cloudsoft.io/ Github: https://github.com/tbouron Twitter: https://twitter.com/eltibouron
