For now, I'm not sure what the solution is, but I would think choosing one specific list of required/supported software (including OS) and documenting that thoroughly is the right start. Something like what currently exists for the vagrant side of things.
That said, long term I would love to see broader support and automatically generating documentation derived from the code. Isn't CentOS 6 still preferred over 7? That's what I've been working on solely for that reason. Jon On Tue, Dec 13, 2016, 21:07 Matt Foley <[email protected]> wrote: > In my work on METRON-608, I’ve found a lot of small but significant bugs > in the existing Mpack (version 0.3.0). These bugs integrate with a lot of > the oddball tweaks specified in the draft Install docs I saw. I would like > to submit a PR for these bugs, but they must be accompanied by changes in > the Install doc. > > How do we want to manage this? Should we version the Install doc? Or > have multiple sections for the different versions? Or have footnotes or > sidebar comments about which version certain paragraphs do and do not apply > to? > > I’m inclined to use sidebars within the document, because the dependencies > aren’t just on which versions of HDP, Ambari, and Metron you use. They > also depend on Python 2.6 vs 2.7, and Centos 6 vs 7. > > Thanks, > --Matt > > > On 12/13/16, 4:13 PM, "[email protected]" <[email protected]> wrote: > > Sorry about the delay. To be honest I delayed once I saw the > management UI > PR because I was going to wait for that to be merged into master > before I > did my mpack install. My initial thought is that there may need to be > some > sort of a merger between what you provided and the Hortonworks blog > post, > as I would prefer a single, comprehensive post over fragmentation. > > Jon > > On Tue, Dec 13, 2016, 18:29 Dima Kovalyov <[email protected]> > wrote: > > > Thank you Jon, > > > > Just wondering if you have finished polishing the document? I will > have > > some time this week to polish and publish it if there is anything > left > > to format. > > > > - Dima > > > > On 12/01/2016 04:41 PM, [email protected] wrote: > > > While I can't delegate access to the wiki, I am working on > migrating the > > > documentation over. I have been modifying the document for > clarity and > > > updating up some of the formatting, so please give me some time to > do so. > > > > > > Jon > > > > > > On Thu, Dec 1, 2016 at 9:33 AM Dima Kovalyov < > [email protected]> > > > wrote: > > > > > >> Hello, > > >> > > >> Who can provide me with an access to add new page in this > Installation > > >> space below? > > >> https://cwiki.apache.org/confluence/display/METRON/Installation > > >> I want to add up-to-date article on how-to install Metron using > Ambari > > >> mpack. > > >> Please advise. > > >> > > >> My username is "dima" registered with electronic mail dimdroll > hosted by > > >> gmail. > > >> > > >> Right now I have it posted here: > > >> https://goo.gl/HWGwpj > > >> But instead of maintaining it there it will be convenient for me > to have > > >> it in confluence instead of gdocs as I am using confluence in our > > internal > > >> project as well, so I can basically copy formatted doc between > each > > other. > > >> > > >> Thank you. > > >> > > >> - Dima > > >> > > >> On 11/30/2016 04:57 PM, [email protected]<mailto:[email protected]> > > wrote: > > >> Probably the best long term place for this would be here< > > >> https://cwiki.apache.org/confluence/display/METRON/Installation>, > > >> potentially with a new mpack article (to differentiate from the > ambari > > >> install which is already there, but dated). I can help with > migrating > > it > > >> to the wiki - I will run through your documentation later today > and > > request > > >> edit permissions if I find anything worth updating. > > >> > > >> Would you be OK if I put your doc in the wiki at some point? > > >> > > >> Jon > > >> > > >> On Wed, Nov 30, 2016 at 9:51 AM Dima Kovalyov < > [email protected] > > >> <mailto:[email protected]>> wrote: > > >> As multiple users asked I have published document for installing > Metron > > on > > >> Centos 6 using Ambari 2.4 with HDP 2.5: > > >> https://goo.gl/HWGwpj > > >> It may look ugly as this is direct export from internal > confluence with > > >> cosmetic changes and our internal details removed. Let me know if > there > > is > > >> a better place to publish it. > > >> > > >> Feel free to request edit permissions, I would like the document > to be > > >> up-to-date, simplified and corrected if there are any errors. The > goal > > for > > >> this document is to provide installation guidance in a manner > that Linux > > >> administrator with basic knowledge could perform installation > without > > >> opening any other links or searching for commands, step by step. > > >> Thank you. > > >> > > >> p.s. feel free to let me know if you will encounter any errors as > I've > > >> seen them bunch. > > >> > > >> - Dima > > >> > > >> > > >> > > >> On 11/30/2016 04:45 PM, [email protected]<mailto:[email protected]> > > wrote: > > >> I also have written up a method to retrieve and setup the required > > >> packages in my script< > > >> > > > https://github.com/JonZeolla/Development/blob/master/Bash/setupMetron.sh> > > >> which could probably be cleaned up and incorporated. > > >> > > >> Jon > > >> > > >> On Wed, Nov 30, 2016 at 9:24 AM Nick Allen <[email protected] > <mailto: > > >> [email protected]>> wrote: > > >> In regards to your comment about a "full package version dump" > there is > > at > > >> least a start at `metron-deployment/scripts/platform-info.sh`. > If we > > can > > >> enhance that, let's do it. > > >> > > >> On Wed, Nov 30, 2016 at 4:04 AM, shoggi <[email protected]<mailto: > > >> [email protected]>> wrote: > > >> Hi > > >> > > >> Just to wrap this up. I've tried to deploy it on HDP 2.5 but I > got the > > >> exact same problem. What I did was removing all components for > 2.4.3.0 > > and > > >> then created a new cluster based on 2.5. I left it at that as the > most > > >> likely issue could be with my base install. May that be wrong > version of > > >> something or even a missing dev package. > > >> > > >> As my work also coincided with the announcement by Casey I went > ahead > > and > > >> installed 0.3.0 as a dev installation instead. That worked great > as the > > >> dev-full-platform guide is very complete. > > >> > > >> As a general feedback, in order to get more people interested and > not > > >> having noobs like me come up with redundant questions - At this > stage > > also > > >> sorry for not providing a meaningful subject, I simply was too > excited > > :) > > >> > > >> There is much to win with a small extension of the installation > guide. I > > >> don't think it needs much more prose around but a full package > version > > dump > > >> with every new release would help greatly to get things going. > For the > > >> things which need to be compiled in, the versioning information > should > > be > > >> enough. I am saying this, as everyone starts with a different base > > system. > > >> Some have the chance to start from scratch, some have a system > running > > and > > >> build something on top and then there are the experienced Metron > users, > > who > > >> have many base requirements.. they tend to forget. And I believe > at the > > end > > >> it always fails just because of some very tiny small difference. > > >> > > >> If there is a space somewhere available, this could even be a > spot, > > where > > >> people can say, "hey, I got this running on XYZ and here is the > package > > >> version dump". A simple table with 'Metron version', 'OS release', > > 'Package > > >> list file', 'Version Details of every other needed to-be-compiled > > package', > > >> is already a big help. This really could supplement the existing > guide > > >> found here: > > >> > > > https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster > > >> . > > >> > > >> But then, I could be alone and others did not experience the same > and > > >> simply got it to run, I don't know. Anyway, I would be happy to > help if > > >> help is needed. > > >> > > >> Cheers > > >> Shoggi > > >> > > >> On Sun, Nov 27, 2016 at 3:00 PM, shoggi <[email protected]<mailto: > > >> [email protected]>> wrote: > > >> Hi Simon > > >> > > >> Thank you for your very quick response. I installed it on HDP > 2.4.3.0. > > >> > > >> I had a 0.2.0 instance running and was very conscious about > versioning > > the > > >> various components. Somehow I had the understanding of not > touching 2.5 > > as > > >> of yet. I give that a try. > > >> > > >> Thanks heaps ! > > >> > > >> Shoggi > > >> > > >> On Sun, Nov 27, 2016 at 2:52 PM, Simon Elliston Ball < > > >> [email protected]<mailto:[email protected]>> > wrote: > > >> Hi Shoggi, > > >> > > >> Can I ask which version of HDP you used as a base. The article is > a > > little > > >> out of date in that is calls for 2.4, which the 0.3.0 release is > based > > on > > >> 2.5. There was a major version difference in storm between the > two, > > which > > >> seems likely to have caused the problem you're seeing. > > >> > > >> Simon > > >> > > >> Sent from my iPad > > >> > > >> On 27 Nov 2016, at 13:48, shoggi <[email protected]<mailto: > > [email protected]>> > > >> wrote: > > >> > > >> Hi all > > >> > > >> I followed the installation guide to deploy Metron via Ambari > > >> > > > https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html > > >> . > > >> > > >> I ended up with all but the Metron Components (Enrichment, > Parsing, > > >> Parsers) to start up. The closest to pinpoint the issue is, when > I try > > to > > >> start any new parser topology such as with bro: > > >> > > >> /usr/metron/0.3.0/bin/start_parser_topology.sh -k > x.localdomain:6667 -z > > >> x.localdomain:2181 -s bro > > >> > > >> The error I get is: > > >> > > >> Exception in thread "main" java.lang.BootstrapMethodError: > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > > >> > > >> Or in more details: > > >> > > >> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= > > >> -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-227/storm > > >> -Dstorm.log.dir=/var/log/storm > > >> > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > > >> -Dstorm.conf.file= -cp > > >> > > > /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/clojure-1.6.0.jar:/usr/hdp/ > > >> 2.4.3.<http://2.4.3 > > >> > > > .>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop-auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/ > > >> 2.4.3.<http://2.4.3 > > >> > > > .>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/ > > >> 2.4.3.<http://2.4.3 > > >> > > > .>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/ > > >> 2.4.3.<http://2.4.3 > > >> > > > .>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/minlog-1.2.jar:/usr/hdp/ > > >> 2.4.3.<http://2.4.3 > > > .>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.3.0-227/storm/bin > > >> -Dstorm.jar=/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar > > >> org.apache.metron.parsers.topology.ParserTopologyCLI -k > > x.localdomain:6667 > > >> -z x.localdomain:2181 -s bro > > >> Exception in thread "main" java.lang.BootstrapMethodError: > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > > >> at > > >> > > > org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<init>(ParserTopologyCLI.java:199) > > >> at > > >> > > > org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<clinit>(ParserTopologyCLI.java:47) > > >> at > > >> > > > org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:266) > > >> Caused by: java.lang.NoClassDefFoundError: org/apache/storm/Config > > >> ... 3 more > > >> Caused by: java.lang.ClassNotFoundException: > org.apache.storm.Config > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > >> ... 3 more > > >> > > >> Must be something wrong with my config, possibly I have overlooked > > >> something but can't put my finger on it. Any help is appreciated. > > >> > > >> Many thanks > > >> Shoggi > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> Nick Allen <[email protected]<mailto:[email protected]>> > > >> -- > > >> > > >> Jon > > >> > > >> Sent from my mobile device > > >> > > >> -- > > >> > > >> Jon > > >> > > >> Sent from my mobile device > > >> > > >> -- > > > Jon > > > > > > Sent from my mobile device > > > > > > > -- > > Jon > > Sent from my mobile device > > > > -- Jon Sent from my mobile device
