David,
No, it’s in my METRON-608 single-node deployment :-)
On 12/14/16, 12:46 PM, "David Lyle" <[email protected]> wrote:
Hi Matt and Jon,
FWIW, Metron with the MPack has been tested extensively on CentOS 7. Works
like a champ. The issue right now is that sensor install is a CentOS
6/Ansible proposition.
I haven't seen the issue with ES that you're experiencing, Matt. Is that in
a 1 Master 3 Data node ES config or something else?
-D...
On Wed, Dec 14, 2016 at 3:29 PM, Matt Foley <[email protected]> wrote:
> I hope we will try to support Centos7. Many of my company’s customers are
> requiring new installs to use RHEL 7 or Centos 7 rather than 6. In
> addition, we have the benefit that Centos 6 RPMs basically always run fine
> in Centos7 (barring heavy-duty low-level system manipulations that Metron,
> Kibana, and ES just don’t use), ES doesn’t distinguish between them, and
> Kibana install support for Centos7 is already in place in the Mpack. And
> of course the Hadoop Stack runs fine in 7.
>
> For what it’s worth, I’ve been working with the Mpack installation on
> Centos7 for the last few weeks, and it works fine except for a persistent
> issue where the Elasticsearch service works fine if launched interactively
> but terminates immediately if launched with same arguments as a service.
> Hope to find the cause in the next couple days. (If anyone knows why,
> would love to hear. The daemonized launch scripts come from ES, not from
> Metron, so “should just work”.)
>
> Thanks,
> --Matt
>
> On 12/13/16, 6:46 PM, "[email protected]" <[email protected]> wrote:
>
> For now, I'm not sure what the solution is, but I would think choosing
> one
> specific list of required/supported software (including OS) and
> documenting
> that thoroughly is the right start. Something like what currently
> exists
> for the vagrant side of things.
>
> That said, long term I would love to see broader support and
> automatically
> generating documentation derived from the code. Isn't CentOS 6 still
> preferred over 7? That's what I've been working on solely for that
> reason.
>
> Jon
>
> On Tue, Dec 13, 2016, 21:07 Matt Foley <[email protected]> wrote:
>
> > In my work on METRON-608, I’ve found a lot of small but significant
> bugs
> > in the existing Mpack (version 0.3.0). These bugs integrate with a
> lot of
> > the oddball tweaks specified in the draft Install docs I saw. I
> would like
> > to submit a PR for these bugs, but they must be accompanied by
> changes in
> > the Install doc.
> >
> > How do we want to manage this? Should we version the Install doc?
> Or
> > have multiple sections for the different versions? Or have
> footnotes or
> > sidebar comments about which version certain paragraphs do and do
> not apply
> > to?
> >
> > I’m inclined to use sidebars within the document, because the
> dependencies
> > aren’t just on which versions of HDP, Ambari, and Metron you use.
> They
> > also depend on Python 2.6 vs 2.7, and Centos 6 vs 7.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/13/16, 4:13 PM, "[email protected]" <[email protected]> wrote:
> >
> > Sorry about the delay. To be honest I delayed once I saw the
> > management UI
> > PR because I was going to wait for that to be merged into master
> > before I
> > did my mpack install. My initial thought is that there may need
> to be
> > some
> > sort of a merger between what you provided and the Hortonworks
> blog
> > post,
> > as I would prefer a single, comprehensive post over
> fragmentation.
> >
> > Jon
> >
> > On Tue, Dec 13, 2016, 18:29 Dima Kovalyov <
> [email protected]>
> > wrote:
> >
> > > Thank you Jon,
> > >
> > > Just wondering if you have finished polishing the document? I
> will
> > have
> > > some time this week to polish and publish it if there is
> anything
> > left
> > > to format.
> > >
> > > - Dima
> > >
> > > On 12/01/2016 04:41 PM, [email protected] wrote:
> > > > While I can't delegate access to the wiki, I am working on
> > migrating the
> > > > documentation over. I have been modifying the document for
> > clarity and
> > > > updating up some of the formatting, so please give me some
> time to
> > do so.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Dec 1, 2016 at 9:33 AM Dima Kovalyov <
> > [email protected]>
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> Who can provide me with an access to add new page in this
> > Installation
> > > >> space below?
> > > >> https://cwiki.apache.org/confluence/display/METRON/
> Installation
> > > >> I want to add up-to-date article on how-to install Metron
> using
> > Ambari
> > > >> mpack.
> > > >> Please advise.
> > > >>
> > > >> My username is "dima" registered with electronic mail
> dimdroll
> > hosted by
> > > >> gmail.
> > > >>
> > > >> Right now I have it posted here:
> > > >> https://goo.gl/HWGwpj
> > > >> But instead of maintaining it there it will be convenient
> for me
> > to have
> > > >> it in confluence instead of gdocs as I am using confluence
> in our
> > > internal
> > > >> project as well, so I can basically copy formatted doc
> between
> > each
> > > other.
> > > >>
> > > >> Thank you.
> > > >>
> > > >> - Dima
> > > >>
> > > >> On 11/30/2016 04:57 PM, [email protected]<mailto:Zeolla
> @GMail.com>
> > > wrote:
> > > >> Probably the best long term place for this would be here<
> > > >> https://cwiki.apache.org/confluence/display/METRON/
> Installation>,
> > > >> potentially with a new mpack article (to differentiate from
> the
> > ambari
> > > >> install which is already there, but dated). I can help
with
> > migrating
> > > it
> > > >> to the wiki - I will run through your documentation later
> today
> > and
> > > request
> > > >> edit permissions if I find anything worth updating.
> > > >>
> > > >> Would you be OK if I put your doc in the wiki at some
point?
> > > >>
> > > >> Jon
> > > >>
> > > >> On Wed, Nov 30, 2016 at 9:51 AM Dima Kovalyov <
> > [email protected]
> > > >> <mailto:[email protected]>> wrote:
> > > >> As multiple users asked I have published document for
> installing
> > Metron
> > > on
> > > >> Centos 6 using Ambari 2.4 with HDP 2.5:
> > > >> https://goo.gl/HWGwpj
> > > >> It may look ugly as this is direct export from internal
> > confluence with
> > > >> cosmetic changes and our internal details removed. Let me
> know if
> > there
> > > is
> > > >> a better place to publish it.
> > > >>
> > > >> Feel free to request edit permissions, I would like the
> document
> > to be
> > > >> up-to-date, simplified and corrected if there are any
> errors. The
> > goal
> > > for
> > > >> this document is to provide installation guidance in a
> manner
> > that Linux
> > > >> administrator with basic knowledge could perform
> installation
> > without
> > > >> opening any other links or searching for commands, step by
> step.
> > > >> Thank you.
> > > >>
> > > >> p.s. feel free to let me know if you will encounter any
> errors as
> > I've
> > > >> seen them bunch.
> > > >>
> > > >> - Dima
> > > >>
> > > >>
> > > >>
> > > >> On 11/30/2016 04:45 PM, [email protected]<mailto:Zeolla
> @GMail.com>
> > > wrote:
> > > >> I also have written up a method to retrieve and setup the
> required
> > > >> packages in my script<
> > > >>
> > >
> > https://github.com/JonZeolla/Development/blob/master/Bash/
> setupMetron.sh>
> > > >> which could probably be cleaned up and incorporated.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Wed, Nov 30, 2016 at 9:24 AM Nick Allen <
> [email protected]
> > <mailto:
> > > >> [email protected]>> wrote:
> > > >> In regards to your comment about a "full package version
> dump"
> > there is
> > > at
> > > >> least a start at `metron-deployment/scripts/
> platform-info.sh`.
> > If we
> > > can
> > > >> enhance that, let's do it.
> > > >>
> > > >> On Wed, Nov 30, 2016 at 4:04 AM, shoggi <[email protected]
> <mailto:
> > > >> [email protected]>> wrote:
> > > >> Hi
> > > >>
> > > >> Just to wrap this up. I've tried to deploy it on HDP 2.5
> but I
> > got the
> > > >> exact same problem. What I did was removing all components
> for
> > 2.4.3.0
> > > and
> > > >> then created a new cluster based on 2.5. I left it at that
> as the
> > most
> > > >> likely issue could be with my base install. May that be
> wrong
> > version of
> > > >> something or even a missing dev package.
> > > >>
> > > >> As my work also coincided with the announcement by Casey I
> went
> > ahead
> > > and
> > > >> installed 0.3.0 as a dev installation instead. That worked
> great
> > as the
> > > >> dev-full-platform guide is very complete.
> > > >>
> > > >> As a general feedback, in order to get more people
> interested and
> > not
> > > >> having noobs like me come up with redundant questions - At
> this
> > stage
> > > also
> > > >> sorry for not providing a meaningful subject, I simply was
> too
> > excited
> > > :)
> > > >>
> > > >> There is much to win with a small extension of the
> installation
> > guide. I
> > > >> don't think it needs much more prose around but a full
> package
> > version
> > > dump
> > > >> with every new release would help greatly to get things
> going.
> > For the
> > > >> things which need to be compiled in, the versioning
> information
> > should
> > > be
> > > >> enough. I am saying this, as everyone starts with a
> different base
> > > system.
> > > >> Some have the chance to start from scratch, some have a
> system
> > running
> > > and
> > > >> build something on top and then there are the experienced
> Metron
> > users,
> > > who
> > > >> have many base requirements.. they tend to forget. And I
> believe
> > at the
> > > end
> > > >> it always fails just because of some very tiny small
> difference.
> > > >>
> > > >> If there is a space somewhere available, this could even be
> a
> > spot,
> > > where
> > > >> people can say, "hey, I got this running on XYZ and here is
> the
> > package
> > > >> version dump". A simple table with 'Metron version', 'OS
> release',
> > > 'Package
> > > >> list file', 'Version Details of every other needed
> to-be-compiled
> > > package',
> > > >> is already a big help. This really could supplement the
> existing
> > guide
> > > >> found here:
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Installation+on+an+Ambari-Managed+Cluster
> > > >> .
> > > >>
> > > >> But then, I could be alone and others did not experience
> the same
> > and
> > > >> simply got it to run, I don't know. Anyway, I would be
> happy to
> > help if
> > > >> help is needed.
> > > >>
> > > >> Cheers
> > > >> Shoggi
> > > >>
> > > >> On Sun, Nov 27, 2016 at 3:00 PM, shoggi <[email protected]
> <mailto:
> > > >> [email protected]>> wrote:
> > > >> Hi Simon
> > > >>
> > > >> Thank you for your very quick response. I installed it on
> HDP
> > 2.4.3.0.
> > > >>
> > > >> I had a 0.2.0 instance running and was very conscious about
> > versioning
> > > the
> > > >> various components. Somehow I had the understanding of not
> > touching 2.5
> > > as
> > > >> of yet. I give that a try.
> > > >>
> > > >> Thanks heaps !
> > > >>
> > > >> Shoggi
> > > >>
> > > >> On Sun, Nov 27, 2016 at 2:52 PM, Simon Elliston Ball <
> > > >> [email protected]<mailto:simon@
> simonellistonball.com>>
> > wrote:
> > > >> Hi Shoggi,
> > > >>
> > > >> Can I ask which version of HDP you used as a base. The
> article is
> > a
> > > little
> > > >> out of date in that is calls for 2.4, which the 0.3.0
> release is
> > based
> > > on
> > > >> 2.5. There was a major version difference in storm between
> the
> > two,
> > > which
> > > >> seems likely to have caused the problem you're seeing.
> > > >>
> > > >> Simon
> > > >>
> > > >> Sent from my iPad
> > > >>
> > > >> On 27 Nov 2016, at 13:48, shoggi <[email protected]<mailto:
> > > [email protected]>>
> > > >> wrote:
> > > >>
> > > >> Hi all
> > > >>
> > > >> I followed the installation guide to deploy Metron via
> Ambari
> > > >>
> > >
> > https://community.hortonworks.com/articles/60805/deploying-
> a-fresh-metron-cluster-using-ambari-serv.html
> > > >> .
> > > >>
> > > >> I ended up with all but the Metron Components (Enrichment,
> > Parsing,
> > > >> Parsers) to start up. The closest to pinpoint the issue is,
> when
> > I try
> > > to
> > > >> start any new parser topology such as with bro:
> > > >>
> > > >> /usr/metron/0.3.0/bin/start_parser_topology.sh -k
> > x.localdomain:6667 -z
> > > >> x.localdomain:2181 -s bro
> > > >>
> > > >> The error I get is:
> > > >>
> > > >> Exception in thread "main" java.lang.BootstrapMethodError:
> > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config
> > > >>
> > > >> Or in more details:
> > > >>
> > > >> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client
> -Ddaemon.name=
> > > >> -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-227/storm
> > > >> -Dstorm.log.dir=/var/log/storm
> > > >>
> > >
> > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/
> lib:/usr/hdp/current/storm-client/lib
> > > >> -Dstorm.conf.file= -cp
> > > >>
> > >
> > /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4.
> 3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> clojure-1.6.0.jar:/usr/hdp/
> > > >> 2.4.3.<http://2.4.3
> > > >>
> > >
> > .>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/
> lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop-
> auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/
> > > >> 2.4.3.<http://2.4.3
> > > >>
> > >
> > .>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/jackson-core-2.3.1.jar:/usr/hdp/
> > > >> 2.4.3.<http://2.4.3
> > > >>
> > >
> > .>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/
> usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/
> hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.
> v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9.
> 94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/
> > > >> 2.4.3.<http://2.4.3
> > > >>
> > >
> > .>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> minlog-1.2.jar:/usr/hdp/
> > > >> 2.4.3.<http://2.4.3
> > >
> > .>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0-
> 227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0-
> 227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0-
> 227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/
> lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/
> storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/
> zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0-
> uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/
> hdp/2.4.3.0-227/storm/bin
> > > >> -Dstorm.jar=/usr/metron/0.3.0/
> lib/metron-parsers-0.3.0-uber.jar
> > > >> org.apache.metron.parsers.topology.ParserTopologyCLI -k
> > > x.localdomain:6667
> > > >> -z x.localdomain:2181 -s bro
> > > >> Exception in thread "main" java.lang.BootstrapMethodError:
> > > >> java.lang.NoClassDefFoundError: org/apache/storm/Config
> > > >> at
> > > >>
> > >
> > org.apache.metron.parsers.topology.ParserTopologyCLI$
> ParserOptions.<init>(ParserTopologyCLI.java:199)
> > > >> at
> > > >>
> > >
> > org.apache.metron.parsers.topology.ParserTopologyCLI$
> ParserOptions.<clinit>(ParserTopologyCLI.java:47)
> > > >> at
> > > >>
> > >
> > org.apache.metron.parsers.topology.ParserTopologyCLI.
> main(ParserTopologyCLI.java:266)
> > > >> Caused by: java.lang.NoClassDefFoundError:
> org/apache/storm/Config
> > > >> ... 3 more
> > > >> Caused by: java.lang.ClassNotFoundException:
> > org.apache.storm.Config
> > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:
> 381)
> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > >> at sun.misc.Launcher$AppClassLoader.loadClass(
> Launcher.java:331)
> > > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > >> ... 3 more
> > > >>
> > > >> Must be something wrong with my config, possibly I have
> overlooked
> > > >> something but can't put my finger on it. Any help is
> appreciated.
> > > >>
> > > >> Many thanks
> > > >> Shoggi
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Nick Allen <[email protected]<mailto:[email protected]>>
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >> Sent from my mobile device
> > > >>
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >> Sent from my mobile device
> > > >>
> > > >> --
> > > > Jon
> > > >
> > > > Sent from my mobile device
> > > >
> > >
> > > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
> >
> >
> > --
>
> Jon
>
> Sent from my mobile device
>
>
>
