Re: Apache Zeppelin
Thanks Michael. I see what you are referring to now. Have a good weekend, Tom. On Fri, 2020-07-17 at 11:47 -0600, Michael Miklavcic wrote: > There are notebooks part of the Ambari install/offering. I don't > think it should break anything core - you could try removing it from > the deps here, along with the config - > https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml#L538 > . There may be other places you'll need to look as well, but that's > probably worth starting from. > > On Fri, Jul 17, 2020 at 9:15 AM Yerex, Tom wrote: > > Good morning, > > > > I would like to work with Apache Zeppelin in our Metron set up. It > > seems there might be some bugs affecting our installation, which > > has > > prompted me to consider removing Zeppelin from Ambari and deploying > > it > > with Ansible. My assumption is by using Ansible, I can avoid > > relying on > > Ambari/mpack and build out our own custom deployment. > > > > An attempt to uninstall Zeppelin (using Ambari), prompts a warning > > message that the dependent service Metron must also be deleted. > > > > I don't see that Metron requires Zeppelin to work. If I can modify > > the > > (Ambari?) configuration files to disable the dependency > > requirement, do > > I risk breaking something in Metron? > > > > Thank you, > > > > Tom. > >
Apache Zeppelin
Good morning, I would like to work with Apache Zeppelin in our Metron set up. It seems there might be some bugs affecting our installation, which has prompted me to consider removing Zeppelin from Ambari and deploying it with Ansible. My assumption is by using Ansible, I can avoid relying on Ambari/mpack and build out our own custom deployment. An attempt to uninstall Zeppelin (using Ambari), prompts a warning message that the dependent service Metron must also be deleted. I don't see that Metron requires Zeppelin to work. If I can modify the (Ambari?) configuration files to disable the dependency requirement, do I risk breaking something in Metron? Thank you, Tom.
RE: Development Activity has dropped to effectively 0, what should we do?
(This is only brainstorming.) I like Metron's documentation. There has been effort and care taken there. Ambari is nice, but given that mpack is moving behind a paywall then it seems that the groups benefiting from the paywall can chip in to build Metron mpacks at their leisure. Elasticsearch is popular so I can see an argument to keep it. On the flip side, Elastisearch is not really trivial to run. Replacing that with a simple template that writes data to a file as an example of how to write an IndexDAO (pardon if my terminology is incorrect), and split Elasticsearch into another repo to be maintained by ELK enthusiasts would reduce the core workload further. Hbase might be another piece that can be put into another project and another simpler example written that relies on something like SQLite could replace it. SQLite is relatively trivial to set up and run. Deployment in Ansible and maintaining the development build, with some work in documentation on how to add modules like (as an example), Elasticsearch and Hbase for "bigger" development work. --T. On 2020-04-21 10:12:52-07:00 Otto Fowler wrote: I think the difference is the maintenance of the core of metron that *has* to be, and other things that may still be done, but will be worked on for their merits or by community need and not be required for everything On April 21, 2020 at 10:29:24, Justin Leet (justinjl...@gmail.com) wrote: How we install depends on what we're choosing to keep around. My concern is getting core Metron's scope down to a supportable level. This entire conversation is probably just a thought experiment until we properly limit the rest of our scope. It's putting the cart before the horse. I want to emphasize this, because we're having a discussion about how to install something that in many ways doesn't actually exist yet. A lot of the install complexity comes from managing so many moving parts at once (ES/Solr, the UI, Kerberos, etc.). If we cut that down, I'm not sure we need a big installer to manage everything. Plenty of projects trust people to be able to run convenience scripts and shell commands. Again, I think this is an academic discussion until we figure out our overall project direction. On Tue, Apr 21, 2020 at 10:02 AM Nick Allen n...@nickallen.org wrote: gt; Hi Tom - gt; gt; gt; Do you or anyone have enough experience to judge if it is possible to gt; leverage Ansible as a replacement to deploy a working cluster? gt; gt; Yes, I worked a lot on the Ansible mechanism in the early days of Metron. gt; This was the primary deployment mechanism before we had the Ambari MPack. gt; gt; We found it very difficult to use Ansible to create a one-size-fits-all gt; deployment solution. It's possible, but very difficult to get a solution gt; that doesn't take close monitoring and manual work arounds when attempting gt; to use it across environments of different sizes and shapes. In terms of gt; usability, the Ambari MPack was a big step-up in my opinion. gt; gt; gt; gt; perhaps a dedicated docker image that is designed to connect with other gt; dockerized applications such as Storm, Kafka, etc..? gt; gt; Yes, I think that would be the way to go for a dev environment. We would be gt; able to use community supported containers for most of our underlying gt; platform needs. Unfortunately, this alone would not help anyone deploy gt; Metron on a cluster. gt; gt; gt; gt; gt; On Tue, Apr 21, 2020 at 9:08 AM Yerex, Tom tom.ye...@ubc.ca wrote: gt; gt; gt; Hi Nick, gt; gt; gt; gt; I see there is a lot of work done using Ansible in the repository. Do you gt; gt; or anyone have enough experience to judge if it is possible to leverage gt; gt; Ansible as a replacement to deploy a working cluster? gt; gt; gt; gt; Now that I am typing this out, I wonder if docker might be a solution gt; that gt; gt; would work? I don't have much experience with docker, perhaps a dedicated gt; gt; docker image that is designed to connect with other dockerized gt; applications gt; gt; such as Storm, Kafka, etc..? gt; gt; gt; gt; --Tom. gt; gt; gt; gt; On 2020-04-17, 11:27 AM, "Nick Allen" n...@nickallen.org wrote: gt; gt; gt; gt; This is a good discussion and one that I haven't fully grappled with gt; gt; in my gt; gt; own mind yet. I'll have more to add, but I just want to chime in on gt; the gt; gt; topic of Ambari at this point. gt; gt; gt; gt; ### Ambari and the Paywall gt; gt; gt; gt; The problem with Ambari is that its installation mechanism requires a gt; gt; repository of compiled packages (RPMs, DEBs, etc.) To install the gt; gt; underlying platform dependencies (like Kafka, HBase, Storm, Zk, etc) gt; we gt; gt; relied on binary packages that were made freely available by gt; gt; Cloudera/Hortonworks. As of this past January, those packages are now gt; gt; behind a paywall. gt; gt; gt; gt; Due to the paywall, installing your own HDP cluster with Ambari is gt; now gt; gt; effectively
Re: Development Activity has dropped to effectively 0, what should we do?
Hi Nick,
I see there is a lot of work done using Ansible in the repository. Do you or
anyone have enough experience to judge if it is possible to leverage Ansible as
a replacement to deploy a working cluster?
Now that I am typing this out, I wonder if docker might be a solution that
would work? I don't have much experience with docker, perhaps a dedicated
docker image that is designed to connect with other dockerized applications
such as Storm, Kafka, etc..?
--Tom.
On 2020-04-17, 11:27 AM, "Nick Allen" wrote:
This is a good discussion and one that I haven't fully grappled with in my
own mind yet. I'll have more to add, but I just want to chime in on the
topic of Ambari at this point.
### Ambari and the Paywall
The problem with Ambari is that its installation mechanism requires a
repository of compiled packages (RPMs, DEBs, etc.) To install the
underlying platform dependencies (like Kafka, HBase, Storm, Zk, etc) we
relied on binary packages that were made freely available by
Cloudera/Hortonworks. As of this past January, those packages are now
behind a paywall.
Due to the paywall, installing your own HDP cluster with Ambari is now
effectively dead. I am not sure if legacy versions of Kafka, HBase, Storm,
etc will continue to be freely available, but even if so, we cannot
continue to rely on this mechanism if new versions and security updates
will not be made available.
The Apache Metron project does not publish compiled binaries or packages
either. We do make the code freely available to allow users to build and
publish their own Metron packages. But even with this capability, unless
you have a means to install the underlying platform dependencies via
Ambari, installing Metron with Ambari has little value.
Unfortunately, I don't see a feasible path forward for Metron's Ambari
MPack.
### Dev Environment
This not only impacts the users of Apache Metron, this impacts contributors
also. Our primary development environment relies on that Ambari MPack. To
continue development on any of the components of Apache Metron, we would
need to build an alternative development environment that can function
despite the paywall. That could take many shapes, but in my opinion it
would be a blocker for continuing any development on Apache Metron,
unfortunately.
Please do let me know if anyone disagrees or can think of an alternative
approach that would allow the current Ambari MPack to remain viable.
On Thu, Apr 16, 2020 at 4:34 PM Dima Kovalyov wrote:
> - Dropping Ambari.
>
> I like the progress that Apache did with Ambari in 2.7. And I don't know a
> better installer/manager for all the services (we use other Hadoop eco
> services besides Metron).
>
> Sometimes its buggy, agents get stuck or server needs reboot from time to
> time, mpacks brake some functionality. But overall I feel this is the
> direction for central management and orchestration.
>
> - Dima
>
> On Wed, Apr 15, 2020, 12:45 Justin Leet wrote:
>
> > This is a bit off the top of my head, but I'd I agree with pretty much
> all
> > of points on what's bringing a lot of overhead. There's probably also a
> > worthwhile discussion about what value we're shooting for the project to
> > provide to people that influences what stays/goes.
> >
> > Thinking out loud a bit
> >
> >- Dropping Storm and moving to Spark drops the very hard to
> >tune/manage/troubleshoot Storm.
> >- Dropping the UIs (and making SQL the external interface) pretty
much
> >implies dropping the REST APIs and ES/Solr. ES/Solr have been a
giant
> >source of dev heartache on the project and they exist primarily for
> the
> >real time use case. People can build whatever UIs or use existing
> tools
> >against Parquet/Hive/whatever.
> >- Dropping Ambari. It's a complex beast to install because of how
many
> >components we have. Dropping the above makes our install much easier
> and
> >should alleviate the need for a complex installer.
> >
> > At that point, we're basically left with
> >
> >- Some Spark for parse -> enrich -> output
> >- The profiler
> >- Stellar
> >- Probably some other misc stuff (sensors, bro kafka plugging, etc.)
> >
> > At a glance, that seems almost an order of magnitude smaller than what
we
> > currently try to handle.
> >
> > I'm not really sure what an appropriate way to handle the profiler is.
>
RE: Centos6 and Centos7 instructions
Thanks Justin. I'll take a crack at it tonight and see how it goes. --Tom. On 2020-04-15 11:32:18-07:00 Justin Leet wrote: I believe you should just have to make a similar README to what's in the Centos6 dir. If you go to the site-book dir and run `mvn site`, you should be able to open the output in your browser and ensure that your changes are there. If I recall correctly, the intent was to eventually drop Centos6 and just use 7. It might be worth revisiting that and just dropping 6, moving the README to 7 and just support 7 going forward. As a note, the page gets deployed at release time with the latest site-book. On Wed, Apr 15, 2020 at 1:24 PM Yerex, Tom tom.ye...@ubc.ca wrote: gt; Good morning, gt; gt; How does one go about updating the documentation at gt; https://metron.apache.org/current-book/metron-deployment/development/centos6/index.html gt; ? gt; gt; I would like to add a similar page for Centos7, which I see is in the gt; repo. Is there any reason to keep the CentOS 6 page, or should it come down? gt; gt; Cheers, gt; gt; Tom. gt; /tom.ye...@ubc.ca
Centos6 and Centos7 instructions
Good morning, How does one go about updating the documentation at https://metron.apache.org/current-book/metron-deployment/development/centos6/index.html? I would like to add a similar page for Centos7, which I see is in the repo. Is there any reason to keep the CentOS 6 page, or should it come down? Cheers, Tom.
Possible approach to solve GeoIP update?
Good afternoon, Reviewing hxxps://issues.apache.org/jira/projects/METRON/issues/METRON-2340 I'm attempting to sketch out a rough solution and I would like guidance from more experienced minds. Maxmind releases code that allows you to build your own mmdb database. The tests I can see in Metron, besides the download, seem to try a few locations like Milton and London. Would it be sufficient to generate geoip databases with those (limited), test locations or alternatively generate geoip databases with hypothetical locations and then host those locations in git for the purpose of the download and testing? The documentation for the settings in Ambari for Metron GeoIP updates could then be slightly updated to add that someone needs to get their own key for GeoIP updates? Cheers, Tom.
RE: Any relation to Spot?
Thanks Zeolla, --T. On 2020-04-09 15:19:58-07:00 zeo...@gmail.com wrote: Nope, different projects with similar goals. Metron came from Cisco OpenSOC and Spot came from ONI. Jon Zeolla On Thu, Apr 9, 2020, 5:57 PM Yerex, Tom tom.ye...@ubc.ca wrote: gt; Good afternoon, gt; gt; I hope everyone is safe and healthy. I tripped across the Apache Spot gt; project while working through some documentation and it made me wonder if gt; there is any relation between Apache Metron and Apache Spot? gt; gt; hxxp://spot.incubator.apache.org/ gt; gt; Cheers, gt; gt; Tom. gt; /tom.ye...@ubc.ca
Any relation to Spot?
Good afternoon, I hope everyone is safe and healthy. I tripped across the Apache Spot project while working through some documentation and it made me wonder if there is any relation between Apache Metron and Apache Spot? hxxp://spot.incubator.apache.org/ Cheers, Tom.
RE: Development Activity has dropped to effectively 0, what should we do?
To me Metron is big and broad in the scope of technology required to get it running. If things were more modular that would go a long way to reducing the learning curve or at least putting it into smaller bites (and it might encourage more people to get involved). If the UI were an add-on module in another project, it would have made it easier for me and it could also encourage my hypothetical buddy who is a web developer expert to get involved since he could focus on the web-ui module instead of trying to tackle all the other pieces that are probably not part of his bailiwick. Stellar is very intriguing, maybe that is not unique to Metron? The architecture of Metron with respect to parsing, enriching, etc., makes a lot of sense to anyone I talk with. These two aspects of Metron seem like standout examples that make for a powerful platform to develop on. Thanks for continuing this discussion, Tom. On 2020-04-08 15:32:46-07:00 Casey Stella wrote: As far as I know there is no minimum bar of development activity to keep a project open. I think we would all be grateful for any investment that you or your organization would want to make. It also occurs to me that your observation is absolutely spot on: we have a LOT of moving parts. I see some deficiencies here: * We depend on a lot of the various hadoop ecosystem projects and they have to work together very precisely: * This makes for a system that is hard to install. * This also makes for a system which is hard to tune/manage * We have a large surface area of coverage * We have an installer, backend system and front-end UI, which stretches our developers a bit thin, especially since there isn't even interest in those systems Perhaps a reconsideration of the scope and technologies that we use would be merited? If we were to decide to, for instance: * Consolidate scope: focus on a viable backend/API rather than a UI * Consolidate technology: reposition ourselves on top of Spark as a consolidated streaming/batch system * Make SQL our external interface: write out to parquet + the Hive metastore and let users pin up presto tables or hive tables as they see fit This might reduce some of our surface area and make it more viable to get started? Anyway, just some thoughts. Casey On Wed, Apr 8, 2020 at 6:20 PM Yerex, Tom mailto:tom.ye...@ubc.ca>> wrote: Hi Casey, I'm new here and new to contributing to an open source project. Thus far my contribution has been questions, however the steep learning curve has had me working to understand all the moving parts for the last 18 months and I see that as a big investment by my organization. What is a level that would be viable? If my organization were to contribute I don't know that it would be soon enough or at the volume that is recognized as viable, which is why I ask the question. On 2020-04-08 15:05:51-07:00 Casey Stella wrote: Hi all, When composing the board report today, I realized that we have effectively had no development in the last quarter on this project. Please be aware that I say this without a shred of blame or judgement (especially so considering I have not contributed in a long time). That being said, I would like to pose the question to the community: Do we feel that this project is viable? If so, how are we going to spur new contributions? If not, then should we begin the process to fold the project? Best, Casey
RE: Development Activity has dropped to effectively 0, what should we do?
Hi Casey, I'm new here and new to contributing to an open source project. Thus far my contribution has been questions, however the steep learning curve has had me working to understand all the moving parts for the last 18 months and I see that as a big investment by my organization. What is a level that would be viable? If my organization were to contribute I don't know that it would be soon enough or at the volume that is recognized as viable, which is why I ask the question. On 2020-04-08 15:05:51-07:00 Casey Stella wrote: Hi all, When composing the board report today, I realized that we have effectively had no development in the last quarter on this project. Please be aware that I say this without a shred of blame or judgement (especially so considering I have not contributed in a long time). That being said, I would like to pose the question to the community: Do we feel that this project is viable? If so, how are we going to spur new contributions? If not, then should we begin the process to fold the project? Best, Casey
Declaring fields as constants in Metron Alerts
Good afternoon, I am considering the work required to move field names that are declared and referenced from within the metron-alerts Angular code to constants that can be set when the project is built. The work appears to be non-trivial, based on what I have gathered so far one approach might be to export a list of constants from environment.ts and environment.prod.ts, which would be used by development, production, and testing code alike. Does anyone have any insight into another approach or suggestions on what I should be looking for? For example, would it make more sense to build an extension to the REST interface that would load the settings dynamically at start up? Thank you, Tom. smime.p7s Description: S/MIME cryptographic signature
Re: [DISCUSS] GeoLite database licensing change, master build broken…..
Hi Otto, Thank you for raising this in the discussion. It seems to me that Maxmind is proactive about providing instructions and code to deliver updates to the local system. I can recall being surprised that the current Metron solution seemed to do more than I expected, i.e., I thought I would need to get Maxmind files into the local file system where Metron would pick those up and load them into HDFS and instead Metron did it all. Perhaps the approach to simplify Metron and have it load files from the local file system into HDFS, how you get the files to the local file system is up to you? On 2020-01-13, 3:52 AM, "Otto Fowler" wrote: https://issues.apache.org/jira/browse/METRON–2340 https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ Maxmind has changed the way the distribute and license the geolite2 database that we use in our builds and distribution. Master build is broken, and users are having issues setting up metron ( https://the-asf.slack.com/archives/CB7Q6AN3T/p1578556024012200) We need to fix the build and figure out how we are going to move on from this. smime.p7s Description: S/MIME cryptographic signature
Re: [DISCUSS] How are you using in Metron?
This is a great topic, thanks for posting it. 1. My team is focussed or about to use 1, 2, 3, 7, 8, 9, 10, 11 (ES), 13, 14. We are in for the long haul and I'm personally very excited to have a (very small), part to play working in Metron. 2. I have only been working with Metron a short time, my initial feeling has been the system feels tightly coupled in some ways, however that might be my own inexperience and other factors. I feel the documentation could be reviewed and improved, not the documentation from commercial groups, rather the user-community driven docs. I find the user community documents are much richer and frankly offer more information when you are prepared to go digging. 4. My own experience is the largest barrier, although I'm working on that. Cheers, Tom. On 2019-10-17, 9:22 AM, "Michael Miklavcic" wrote: I'd like to kick off a discussion to get a sense of how the broader community is currently using Metron. 1. What features are you using or seriously considering? e.g. 1. enrichments 2. streaming enrichments 3. profiler 4. pcap 5. flatfile summarizer 6. MaaS 7. Meta alerts 8. parser aggregation 9. config UI 10. alert UI 11. solr, ES 12. Stellar 13. Stellar REPL 14. REST API 15. other? 2. What features would you like to see added or improved? 3. What features do you consider to be core to Metron as a platform? 4. If you're using Metron, but not an active community contributor, what would it take to get you more involved in the project? We are close to finishing up a feature branch around upgrading to HDP 3.1, and subsequently on the doorstep of a 1.0 release. This is a huge milestone for the project. I think it's time to take some lessons learned over the past several years and consider what the next phase of Metron will be. Whether you've participated in community discussions before or not, we'd love to hear from you. Best, Mike Miklavcic PMC Apache Metron smime.p7s Description: S/MIME cryptographic signature
Contributing to Metron
I would like to get access to JIRA (or be assigned a ticket), to add documentation to the Centos7 project folder under metron/metron-deployment/development/centos7/, similar to the README.md file that is under the centos6 folder. I am new to open source development, hopefully this is a simple enough change that I will not cause too much of a distraction and contribute to the project. Thank you, Tom. -- Tom Yerex Cybersecurity Analyst Cybersecurity | CISO Office The University of British Columbia | Musqueam Traditional Territory Ponderosa Office Annex A | Vancouver BC | V6T1Z2 Canada Phone 604 822 6531 Privacy Matters @ UBC smime.p7s Description: S/MIME cryptographic signature
Re: Slack channel access
I received the invite and I am signed up. Thank you Michael. -- Tom Yerex Cybersecurity Analyst Phone 604 822 6531 Privacy Matters @ UBC On 2019-03-28, 3:19 PM, "Michael Miklavcic" wrote: Hi Tom, check your inbox for the invite. On Thu, Mar 28, 2019 at 3:37 PM Yerex, Tom wrote: > Good afternoon, > > > > Please add me to the Metron Slack Channel. > > > > Thank you. > > > smime.p7s Description: S/MIME cryptographic signature
Slack channel access
Good afternoon, Please add me to the Metron Slack Channel. Thank you. smime.p7s Description: S/MIME cryptographic signature
