Hi Matt and Jon, FWIW, Metron with the MPack has been tested extensively on CentOS 7. Works like a champ. The issue right now is that sensor install is a CentOS 6/Ansible proposition.
I haven't seen the issue with ES that you're experiencing, Matt. Is that in a 1 Master 3 Data node ES config or something else? -D... On Wed, Dec 14, 2016 at 3:29 PM, Matt Foley <mfo...@hortonworks.com> wrote: > I hope we will try to support Centos7. Many of my company’s customers are > requiring new installs to use RHEL 7 or Centos 7 rather than 6. In > addition, we have the benefit that Centos 6 RPMs basically always run fine > in Centos7 (barring heavy-duty low-level system manipulations that Metron, > Kibana, and ES just don’t use), ES doesn’t distinguish between them, and > Kibana install support for Centos7 is already in place in the Mpack. And > of course the Hadoop Stack runs fine in 7. > > For what it’s worth, I’ve been working with the Mpack installation on > Centos7 for the last few weeks, and it works fine except for a persistent > issue where the Elasticsearch service works fine if launched interactively > but terminates immediately if launched with same arguments as a service. > Hope to find the cause in the next couple days. (If anyone knows why, > would love to hear. The daemonized launch scripts come from ES, not from > Metron, so “should just work”.) > > Thanks, > --Matt > > On 12/13/16, 6:46 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote: > > For now, I'm not sure what the solution is, but I would think choosing > one > specific list of required/supported software (including OS) and > documenting > that thoroughly is the right start. Something like what currently > exists > for the vagrant side of things. > > That said, long term I would love to see broader support and > automatically > generating documentation derived from the code. Isn't CentOS 6 still > preferred over 7? That's what I've been working on solely for that > reason. > > Jon > > On Tue, Dec 13, 2016, 21:07 Matt Foley <ma...@apache.org> wrote: > > > In my work on METRON-608, I’ve found a lot of small but significant > bugs > > in the existing Mpack (version 0.3.0). These bugs integrate with a > lot of > > the oddball tweaks specified in the draft Install docs I saw. I > would like > > to submit a PR for these bugs, but they must be accompanied by > changes in > > the Install doc. > > > > How do we want to manage this? Should we version the Install doc? > Or > > have multiple sections for the different versions? Or have > footnotes or > > sidebar comments about which version certain paragraphs do and do > not apply > > to? > > > > I’m inclined to use sidebars within the document, because the > dependencies > > aren’t just on which versions of HDP, Ambari, and Metron you use. > They > > also depend on Python 2.6 vs 2.7, and Centos 6 vs 7. > > > > Thanks, > > --Matt > > > > > > On 12/13/16, 4:13 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote: > > > > Sorry about the delay. To be honest I delayed once I saw the > > management UI > > PR because I was going to wait for that to be merged into master > > before I > > did my mpack install. My initial thought is that there may need > to be > > some > > sort of a merger between what you provided and the Hortonworks > blog > > post, > > as I would prefer a single, comprehensive post over > fragmentation. > > > > Jon > > > > On Tue, Dec 13, 2016, 18:29 Dima Kovalyov < > dima.koval...@sstech.us> > > wrote: > > > > > Thank you Jon, > > > > > > Just wondering if you have finished polishing the document? I > will > > have > > > some time this week to polish and publish it if there is > anything > > left > > > to format. > > > > > > - Dima > > > > > > On 12/01/2016 04:41 PM, zeo...@gmail.com wrote: > > > > While I can't delegate access to the wiki, I am working on > > migrating the > > > > documentation over. I have been modifying the document for > > clarity and > > > > updating up some of the formatting, so please give me some > time to > > do so. > > > > > > > > Jon > > > > > > > > On Thu, Dec 1, 2016 at 9:33 AM Dima Kovalyov < > > dima.koval...@sstech.us> > > > > wrote: > > > > > > > >> Hello, > > > >> > > > >> Who can provide me with an access to add new page in this > > Installation > > > >> space below? > > > >> https://cwiki.apache.org/confluence/display/METRON/ > Installation > > > >> I want to add up-to-date article on how-to install Metron > using > > Ambari > > > >> mpack. > > > >> Please advise. > > > >> > > > >> My username is "dima" registered with electronic mail > dimdroll > > hosted by > > > >> gmail. > > > >> > > > >> Right now I have it posted here: > > > >> https://goo.gl/HWGwpj > > > >> But instead of maintaining it there it will be convenient > for me > > to have > > > >> it in confluence instead of gdocs as I am using confluence > in our > > > internal > > > >> project as well, so I can basically copy formatted doc > between > > each > > > other. > > > >> > > > >> Thank you. > > > >> > > > >> - Dima > > > >> > > > >> On 11/30/2016 04:57 PM, zeo...@gmail.com<mailto:Zeolla > @GMail.com> > > > wrote: > > > >> Probably the best long term place for this would be here< > > > >> https://cwiki.apache.org/confluence/display/METRON/ > Installation>, > > > >> potentially with a new mpack article (to differentiate from > the > > ambari > > > >> install which is already there, but dated). I can help with > > migrating > > > it > > > >> to the wiki - I will run through your documentation later > today > > and > > > request > > > >> edit permissions if I find anything worth updating. > > > >> > > > >> Would you be OK if I put your doc in the wiki at some point? > > > >> > > > >> Jon > > > >> > > > >> On Wed, Nov 30, 2016 at 9:51 AM Dima Kovalyov < > > dima.koval...@sstech.us > > > >> <mailto:dima.koval...@sstech.us>> wrote: > > > >> As multiple users asked I have published document for > installing > > Metron > > > on > > > >> Centos 6 using Ambari 2.4 with HDP 2.5: > > > >> https://goo.gl/HWGwpj > > > >> It may look ugly as this is direct export from internal > > confluence with > > > >> cosmetic changes and our internal details removed. Let me > know if > > there > > > is > > > >> a better place to publish it. > > > >> > > > >> Feel free to request edit permissions, I would like the > document > > to be > > > >> up-to-date, simplified and corrected if there are any > errors. The > > goal > > > for > > > >> this document is to provide installation guidance in a > manner > > that Linux > > > >> administrator with basic knowledge could perform > installation > > without > > > >> opening any other links or searching for commands, step by > step. > > > >> Thank you. > > > >> > > > >> p.s. feel free to let me know if you will encounter any > errors as > > I've > > > >> seen them bunch. > > > >> > > > >> - Dima > > > >> > > > >> > > > >> > > > >> On 11/30/2016 04:45 PM, zeo...@gmail.com<mailto:Zeolla > @GMail.com> > > > wrote: > > > >> I also have written up a method to retrieve and setup the > required > > > >> packages in my script< > > > >> > > > > > https://github.com/JonZeolla/Development/blob/master/Bash/ > setupMetron.sh> > > > >> which could probably be cleaned up and incorporated. > > > >> > > > >> Jon > > > >> > > > >> On Wed, Nov 30, 2016 at 9:24 AM Nick Allen < > n...@nickallen.org > > <mailto: > > > >> n...@nickallen.org>> wrote: > > > >> In regards to your comment about a "full package version > dump" > > there is > > > at > > > >> least a start at `metron-deployment/scripts/ > platform-info.sh`. > > If we > > > can > > > >> enhance that, let's do it. > > > >> > > > >> On Wed, Nov 30, 2016 at 4:04 AM, shoggi <sho...@gmail.com > <mailto: > > > >> sho...@gmail.com>> wrote: > > > >> Hi > > > >> > > > >> Just to wrap this up. I've tried to deploy it on HDP 2.5 > but I > > got the > > > >> exact same problem. What I did was removing all components > for > > 2.4.3.0 > > > and > > > >> then created a new cluster based on 2.5. I left it at that > as the > > most > > > >> likely issue could be with my base install. May that be > wrong > > version of > > > >> something or even a missing dev package. > > > >> > > > >> As my work also coincided with the announcement by Casey I > went > > ahead > > > and > > > >> installed 0.3.0 as a dev installation instead. That worked > great > > as the > > > >> dev-full-platform guide is very complete. > > > >> > > > >> As a general feedback, in order to get more people > interested and > > not > > > >> having noobs like me come up with redundant questions - At > this > > stage > > > also > > > >> sorry for not providing a meaningful subject, I simply was > too > > excited > > > :) > > > >> > > > >> There is much to win with a small extension of the > installation > > guide. I > > > >> don't think it needs much more prose around but a full > package > > version > > > dump > > > >> with every new release would help greatly to get things > going. > > For the > > > >> things which need to be compiled in, the versioning > information > > should > > > be > > > >> enough. I am saying this, as everyone starts with a > different base > > > system. > > > >> Some have the chance to start from scratch, some have a > system > > running > > > and > > > >> build something on top and then there are the experienced > Metron > > users, > > > who > > > >> have many base requirements.. they tend to forget. And I > believe > > at the > > > end > > > >> it always fails just because of some very tiny small > difference. > > > >> > > > >> If there is a space somewhere available, this could even be > a > > spot, > > > where > > > >> people can say, "hey, I got this running on XYZ and here is > the > > package > > > >> version dump". A simple table with 'Metron version', 'OS > release', > > > 'Package > > > >> list file', 'Version Details of every other needed > to-be-compiled > > > package', > > > >> is already a big help. This really could supplement the > existing > > guide > > > >> found here: > > > >> > > > > > https://cwiki.apache.org/confluence/display/METRON/ > Metron+Installation+on+an+Ambari-Managed+Cluster > > > >> . > > > >> > > > >> But then, I could be alone and others did not experience > the same > > and > > > >> simply got it to run, I don't know. Anyway, I would be > happy to > > help if > > > >> help is needed. > > > >> > > > >> Cheers > > > >> Shoggi > > > >> > > > >> On Sun, Nov 27, 2016 at 3:00 PM, shoggi <sho...@gmail.com > <mailto: > > > >> sho...@gmail.com>> wrote: > > > >> Hi Simon > > > >> > > > >> Thank you for your very quick response. I installed it on > HDP > > 2.4.3.0. > > > >> > > > >> I had a 0.2.0 instance running and was very conscious about > > versioning > > > the > > > >> various components. Somehow I had the understanding of not > > touching 2.5 > > > as > > > >> of yet. I give that a try. > > > >> > > > >> Thanks heaps ! > > > >> > > > >> Shoggi > > > >> > > > >> On Sun, Nov 27, 2016 at 2:52 PM, Simon Elliston Ball < > > > >> si...@simonellistonball.com<mailto:simon@ > simonellistonball.com>> > > wrote: > > > >> Hi Shoggi, > > > >> > > > >> Can I ask which version of HDP you used as a base. The > article is > > a > > > little > > > >> out of date in that is calls for 2.4, which the 0.3.0 > release is > > based > > > on > > > >> 2.5. There was a major version difference in storm between > the > > two, > > > which > > > >> seems likely to have caused the problem you're seeing. > > > >> > > > >> Simon > > > >> > > > >> Sent from my iPad > > > >> > > > >> On 27 Nov 2016, at 13:48, shoggi <sho...@gmail.com<mailto: > > > sho...@gmail.com>> > > > >> wrote: > > > >> > > > >> Hi all > > > >> > > > >> I followed the installation guide to deploy Metron via > Ambari > > > >> > > > > > https://community.hortonworks.com/articles/60805/deploying- > a-fresh-metron-cluster-using-ambari-serv.html > > > >> . > > > >> > > > >> I ended up with all but the Metron Components (Enrichment, > > Parsing, > > > >> Parsers) to start up. The closest to pinpoint the issue is, > when > > I try > > > to > > > >> start any new parser topology such as with bro: > > > >> > > > >> /usr/metron/0.3.0/bin/start_parser_topology.sh -k > > x.localdomain:6667 -z > > > >> x.localdomain:2181 -s bro > > > >> > > > >> The error I get is: > > > >> > > > >> Exception in thread "main" java.lang.BootstrapMethodError: > > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > > > >> > > > >> Or in more details: > > > >> > > > >> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client > -Ddaemon.name= > > > >> -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-227/storm > > > >> -Dstorm.log.dir=/var/log/storm > > > >> > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/ > lib:/usr/hdp/current/storm-client/lib > > > >> -Dstorm.conf.file= -cp > > > >> > > > > > /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4. > 3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > clojure-1.6.0.jar:/usr/hdp/ > > > >> 2.4.3.<http://2.4.3 > > > >> > > > > > .>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/ > lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop- > auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/ > > > >> 2.4.3.<http://2.4.3 > > > >> > > > > > .>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/jackson-core-2.3.1.jar:/usr/hdp/ > > > >> 2.4.3.<http://2.4.3 > > > >> > > > > > .>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/ > usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/ > hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0. > v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9. > 94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/ > > > >> 2.4.3.<http://2.4.3 > > > >> > > > > > .>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > minlog-1.2.jar:/usr/hdp/ > > > >> 2.4.3.<http://2.4.3 > > > > > .>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0- > 227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0- > 227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0- > 227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/ > lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/ > storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/ > zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0- > uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/ > hdp/2.4.3.0-227/storm/bin > > > >> -Dstorm.jar=/usr/metron/0.3.0/ > lib/metron-parsers-0.3.0-uber.jar > > > >> org.apache.metron.parsers.topology.ParserTopologyCLI -k > > > x.localdomain:6667 > > > >> -z x.localdomain:2181 -s bro > > > >> Exception in thread "main" java.lang.BootstrapMethodError: > > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > > > >> at > > > >> > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI$ > ParserOptions.<init>(ParserTopologyCLI.java:199) > > > >> at > > > >> > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI$ > ParserOptions.<clinit>(ParserTopologyCLI.java:47) > > > >> at > > > >> > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI. > main(ParserTopologyCLI.java:266) > > > >> Caused by: java.lang.NoClassDefFoundError: > org/apache/storm/Config > > > >> ... 3 more > > > >> Caused by: java.lang.ClassNotFoundException: > > org.apache.storm.Config > > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java: > 381) > > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > >> at sun.misc.Launcher$AppClassLoader.loadClass( > Launcher.java:331) > > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > >> ... 3 more > > > >> > > > >> Must be something wrong with my config, possibly I have > overlooked > > > >> something but can't put my finger on it. Any help is > appreciated. > > > >> > > > >> Many thanks > > > >> Shoggi > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> -- > > > >> Nick Allen <n...@nickallen.org<mailto:n...@nickallen.org>> > > > >> -- > > > >> > > > >> Jon > > > >> > > > >> Sent from my mobile device > > > >> > > > >> -- > > > >> > > > >> Jon > > > >> > > > >> Sent from my mobile device > > > >> > > > >> -- > > > > Jon > > > > > > > > Sent from my mobile device > > > > > > > > > > -- > > > > Jon > > > > Sent from my mobile device > > > > > > > > -- > > Jon > > Sent from my mobile device > > >
