[ANNOUNCE] Apache Metron moved to Apache Attic

[Justin Leet]

A heads up for the Metron user community that the Metron project has
been 'moved to the Attic'. This means that the Metron developers (more
formally its Project Management Committee) have voted to retire Metron
and move the responsibility for its oversight over to the Attic project.

Loosely speaking this means that the project's resources will be moved to a
read-only state.

You can read more about the Apache Attic and the process of moving to the
Attic at http://attic.apache.org.

You can follow this process in JIRA:

  https://issues.apache.org/jira/browse/ATTIC-190

Thanks,

Justin Leet on behalf of Metron + the Attic.

Re: Drop events from Metron parser

[Justin Leet]

At the parser level, there's some configuration you can use for filtering
events. Specifically "filterClassName".  Take a look at the documentation,
you can either use a custom class, or use Stellar.  The example is even for
"exists(field)", which you could modify to fail for missing fields.

https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html


On Tue, May 5, 2020 at 7:53 PM Yerex, Tom  wrote:

> Good afternoon,
>
> Our incoming data is not always perfect, in some cases events are simply
> missing fields. We would like a way to drop events when particular fields
> are empty (or have values we don't care about).
>
> One way we thought to do this might be to write a custom Stellar function.
> Does anyone know of another solution?
>
> Thank you,
>
> Tom.
>

[ANNOUNCE] Apache Metron-bro-plugin-kafka release 0.3.0

[Justin Leet]

Hi all,

I’m pleased to announce the release of Metron 0.3.0!  It's been a little
while coming, but there's a good number of improvements and fixes, both
around functionality and testing.  Thanks to everyone who's contributing to
and using the plugin!

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/metron-bro-plugin-kafka/0.3.0/

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/metron-bro-plugin-kafka/0.3.0/

The release branch in GitHub is
https://github.com/apache/metron-bro-plugin-kafka/tree/Metron-bro-plugin-kafka_0.3.0
(tag
apache-metron-bro-plugin-kafka_0.3.0-release)

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-2269 Cannot run Docker tests if src is not a git repo
(ottobackwards) closes apache/metron-bro-plugin-kafka#37
METRON-2069 Add btests for bro plugin topic_name selection
(ottobackwards) closes apache/metron-bro-plugin-kafka#36
METRON-2045 Pass a version argument to the bro plugin docker
scripts (JonZeolla) closes apache/metron-bro-plugin-kafka#35
METRON-2003 Bro plugin topic should fall back to the log writer's
path (JonZeolla) closes apache/metron-bro-plugin-kafka#26
METRON-2025 Bro Kafka Plugin Docker should yum clean
(ottobackwards) closes apache/metron-bro-plugin-kafka#33
METRON-2021 Add screen to bro docker image (ottobackwards) closes
apache/metron-bro-plugin-kafka#32
METRON-2013 The bro plugin docker scripts topic name should
be configurable (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#27
METRON-2020 Running run_end_to_end.sh with docker give warning if
bash  4.0 (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#31
METRON-1991 Bro plugin docker scripts should exit nonzero when bro
and kafka counts differ (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#29
METRON-2017 The Bro plugin docker data processing script
incorrectly runs bro (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#30
METRON-1990 Bro plugin docker should exit nonzero if it encounters
issues (JonZeolla) closes apache/metron-bro-plugin-kafka#28
METRON-2004 Bro plugin kafka docker_execute_shell.sh workdir
should be unspecified (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#25
METRON-2000 Fix bro plugin docker line counting for BRO_COUNT
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#24
METRON-1992 Support sending a log to multiple topics (JonZeolla)
closes apache/metron-bro-plugin-kafka#23
METRON-1910 bro plugin segfaults on src/KafkaWriter.cc:72
(JonZeolla) closes apache/metron-bro-plugin-kafka#20
METRON-1911 Create Docker based test environment for Bro Kafka
Plugin (ottobackwards) closes apache/metron-bro-plugin-kafka#21
METRON-1885 Remove version from bro plugin btest (JonZeolla)
closes apache/metron-bro-plugin-kafka#19
METRON-1827 Update librdkafka in metron-bro-plugin-kafka
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#13
METRON-1866 Improve metron-bro-plugin-kafka documentation
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#17
METRON-1304 Allow metron-bro-plugin-kafka to include or exclude
logs (JonZeolla via nickwallen) closes
apache/metron-bro-plugin-kafka#2
METRON-1865 Fix metron-bro-plugin-kafka tests (JonZeolla via
jonzeolla) closes apache/metron-bro-plugin-kafka#16
METRON-1828 Improve bro plugin contributing documentation
(JonZeolla) closes apache/metron-bro-plugin-kafka#14
METRON-1818 Remove config_files from bro-pkg.meta (JonZeolla)
closes apache/metron-bro-plugin-kafka#11
METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla
via jonzeolla) closes apache/metron-bro-plugin-kafka#10
METRON-1773 Bro plugin docs should refer to Apache Metron project
(nickwallen) closes apache/metron-bro-plugin-kafka#9

Re: Central Navigation Use Case

[Justin Leet]

This feels like our personas should correspond to roles and the navigation
available potentially based on that.

Especially for smaller groups and POC type stuff, I'd expect there's people
with overlapping personas.

At that point, central nav either displays available UIs (especially if we
add more), or nothing if there's just one option.

On Mon, Mar 11, 2019, 06:32 Shane Ardell  wrote:

> Hello all,
>
> Currently, Metron has two distinct UIs separated from one another. The
> idea behind this original design choice was to separate the UIs based on
> user profiles; the Alerts UI for SOC analysts vs the Management UI for
> Operations personnel.
>
> However, are there any users out there that use both screens? Is there any
> interests in having a central navigation to make it easier to switch back
> and forth between both UIs?
>
> Thanks in advance,
> Shane
>
>

[ANNOUNCE] Apache Metron release 0.7.0

[Justin Leet]

Hi all,

I’m pleased to announce the release of Metron 0.7.0! There's been a lot of
work on improvements, upgrades, discussion, and more. Thanks to everyone
who's contributed, and thank you to our users.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.7.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.7.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.7.0 (tag
apache-metron_0.7.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet)
closes apache/metron#1293
METRON-1931 Update dev utilities to support new repo location
(rlenferink via justinleet) closes apache/metron#1295
METRON-1922 Escaping incorrectly handled in current aesh version
(justinleet) closes apache/metron#1291
METRON-1867 Remove `/api/v1/update/replace` endpoint (nickwallen)
closes apache/metron#1284
METRON-1810 Storm Profiler Intermittent Test Failure (nickwallen)
closes apache/metron#1289
METRON-1909 Remove http filter from release utils changelog
generation (justinleet) closes apache/metron#1283
METRON-1869 Unable to Sort an Escalated Meta Alert (nickwallen)
closes apache/metron#1280
METRON-1889: Add any missing timestamp fields to unified
enrichment topology (mmiklavc via mmiklavc) closes apache/metron#1286
METRON-1913 metron-alert UI - Build broken by missing transitive
dependency (tiborm via sardell) closes apache/metron#1285
METRON-1845 Correct Test Data Load in Elasticsearch Integration
Tests (nickwallen) closes apache/metron#1247
METRON-1888 Default Topology Settings in MPack Cause Profiler to
Stall (nickwallen) closes apache/metron#1276
METRON-1887: Add logging to the ClasspathFunctionResolver
(mmiklavc via mmiklavc) closes apache/metron#1274
METRON-1873 Update Bootstrap version in Management UI (sardell)
closes apache/metron#1267
METRON-1825 Upgrade bro to 2.5.5 (JonZeolla via nickwallen) closes
apache/metron#1237
METRON-1890 Metron Vagrant should disable audio (ottobackwards)
closes apache/metron#1277
METRON-1874 Create a Parser Debugger (nickwallen) closes apache/metron#1265
METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
apache/metron#1270
METRON-1877 Nested IF ELSE statements can cause parse errors in
Stellar (justinleet) closes apache/metron#1268
METRON-1872 Move rat plugin away from snapshot version
(justinleet) closes apache/metron#1264
METRON-1875 Expose configurable global settings in the Alerts UI
(merrimanr) closes apache/metron#1266
METRON-1834: Migrate Elasticsearch from TransportClient to new
Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
METRON-1834: Migrate Elasticsearch from TransportClient to new
Java REST API (cstella via mmiklavc)
METRON-1749 Update Angular to latest release in Management UI
(sardell via nickwallen) closes apache/metron#1217
METRON-1870 Intermittent Stellar REST test failures (merrimanr via
nickwallen) closes apache/metron#1263
METRON-1868 metron-committer-common incorrectly checking REPO_NAME
(JonZeolla via jonzeolla) closes apache/metron#1260
METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM
syslog messages (liuy-tnz via nickwallen) closes apache/metron#1171
METRON-1847 Create reusable script with functions from
prepare-commit (ottobackwards) closes apache/metron#1248
METRON-1850 Stellar REST function (merrimanr) closes apache/metron#1250
METRON-1858 BasicFireEyeParser check style cleanup and
optimization (ottobackwards) closes apache/metron#1255
METRON-1864 Stellar date format test fails after daylight saving
(ottobackwards) closes apache/metron#1258
METRON-1861 METRON-1861: REST fails to start when LDAP enabled and
'Active Spring profiles' config is empty (anandsubbu via justinleet)
closes apache/metron#1256
METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver
(mmiklavc via mmiklavc) closes apache/metron#1251
METRON-1857 Fix Metaalert Nested Alert Field Name in Index
Template (nickwallen) closes apache/metron#1253
METRON-1855: Make unified enrichment topology the default and
deprecate split-join (mmiklavc via mmiklavc) closes apache/metron#1252
METRON-1790 Unsubscribe from every observable in the pcap panel UI
component (ruffle via nickwallen) closes apache/metron#1208
METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc)

[ANNOUNCE] Apache Metron release 0.6.0

[Justin Leet]

Hi All,

I’m happy to announce the release of Metron 0.6.0! There's a been a lot of
great work everywhere on the project, and thanks to both everyone who
contributed and our users.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.6.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.6.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.6.0 (tag
apache-metron-0.6.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

Metron CHANGES (in reverse chronological order):

METRON-1764 Update version to 0.6.0 (justinleet) closes apache/metron#1183
METRON-1751 Storm Profiler dies when consuming null message
(nickwallen) closes apache/metron#1176
METRON-1757 Storm Profiler Serialization Exception (nickwallen)
closes apache/metron#1178
METRON-1743 CEF testPaloAltoCEF test using a confusing variable
name (JonZeolla via justinleet) closes apache/metron#1173
METRON-1752 Prevent package.lock from changing during build
(sardell via merrimanr) closes apache/metron#1177
METRON-1724 Date/time validation missing in PCAP query (tiborm via
nickwallen) closes apache/metron#1172
METRON-1739 UDP packets are not handled (merrimanr) closes
apache/metron#1168
METRON-1727: Alerts are not populated on the alerts UI after
enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
apache/metron#1141
METRON-1738: Pcap directories should have correct permissions
(merrimanr via mmiklavc) closes apache/metron#1166
METRON-1737: Document Job cleanup (merrimanr via mmiklavc) closes
apache/metron#1164
METRON-1732: Fix job status liveness bug and parallelize finalizer
file writing (mmiklavc via mmiklavc) closes apache/metron#1157
METRON-1735 Empty print status option causes NPE (merrimanr)
closes apache/metron#1160
METRON-1733 PCAP UI - PCAP queries dont work on Safari
(sardell via merrimanr) closes apache/metron#1158
METRON-1734 Src and Dst port filters are incorrect after changing
to empty (merrimanr) closes apache/metron#1159
METRON-1725 Add ability to specify YARN queue for pcap jobs
(merrimanr) closes apache/metron#1153
METRON-1731: PCAP - Escape colons in output dir names (mmiklavc
via mmiklavc) closes apache/metron#1155
METRON-1702 Reload a running job in the UI (merrimanr) closes
apache/metron#1156
METRON-1722 PcapCLI should print progress to stdout (merrimanr)
closes apache/metron#1138
METRON-1728: Handle null values in config in Pcap backend more
gracefully (mmiklavc via mmiklavc) closes apache/metron#1151
METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc via
mmiklavc) closes apache/metron#1152
METRON-1713 PCAP UI - Add a way to kill a pcap job (tiborm via
merrimanr) closes apache/metron#1143
METRON-1723 PCAP UI - Unable to select/copy from packets details
in PCAP query panel (sardell via merrimanr) closes apache/metron#1139
METRON-1712 PCAP UI - Input validation (tiborm via merrimanr)
closes apache/metron#1142
METRON-1720 Better error messages when there are no results or
wireshark is not installed (merrimanr) closes apache/metron#1154
METRON-1726: Refactor PcapTopologyIntegrationTest (mmiklavc via
mmiklavc) closes apache/metron#1140
METRON-1683 PCAP UI - Fix the download progress bar (sardell via
merrimanr) closes apache/metron#1122
METRON-1675 PCAP UI - Introduce the paging capability (sardell via
merrimanr) closes apache/metron#1121
METRON-1721 New default input path is wrong in pcap CLI
(merrimanr) closes apache/metron#1137
METRON-1676 PCAP UI - Add data range selector to the filter bar
(tiborm via merrimanr) closes apache/metron#1119
METRON-1662 PCAP UI - Downloading PCAP page files (tiborm via
merrimanr) closes apache/metron#1118
METRON-1700 Create REST endpoint to get job configuration
(merrimanr) closes apache/metron#1135
METRON-1671 Create PCAP UI (tiborm via merrimanr) closes apache/metron#1103
METRON-1701 Update General notes on the installation of Pycapa on
Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
METRON-1650 Packaging docker containers are too large (jameslamb
via merrimanr) closes apache/metron#1091
METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
management pack repo info closes apache/incubator-metron#1052
METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
apache/incubator-metron#1126
METRON-1694: Clean up Metron REST docs closes apache/incubator-metron#1131

Re: Google Cloud Platform

[Justin Leet]

Unfortunately, I have no familiarity with GCP at all, but a good place to
start *may* be by reverse engineering some of our EC2 instructions
.
You might be able to sub in GCP steps as needed for provisioning and more
or less follow the internal instructions otherwise.  Keep in mind Metron
itself is deployed via Ambari, so as long as you can get a Hadoop cluster
up and running, the RPMs out and installed + the mpack, you should at least
be able to take a good stab at getting things up and running.

I'd be curious if anyone has any GCP experience at all and would know if
this is a reasonable approach.

If you do make an attempt, I'd definitely like to hear back on how it goes,
and what issues where hit, etc.

Justin

On Thu, Aug 9, 2018 at 1:04 AM Kevin Waterson 
wrote:

> Was hoping somebody else had.. not sure where to start... :)
>
>
> On Thu, Aug 9, 2018 at 2:00 AM James Sirota  wrote:
>
>> Not to my knowledge. Are you trying it?
>>
>>
>> 24.07.2018, 22:19, "Kevin Waterson" :
>>
>> Has anybody been able to deploy Metron using GCP?
>>
>> Thanks
>> Kevin
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>

[ANNOUNCE] Apache Metron release 0.5.0

[Justin Leet]

Hi All,

I’m happy to announce the release of Metron 0.5.0!  Everyone has put in a
lot of working into improvements, new features, and discussion.  Thanks to
everyone who contributed, and I look forward to having users enjoy our new
features and improvements.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.5.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.5.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.5.0 (tag
apache-metron-0.5.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

Metron CHANGES (in reverse chronological order):

METRON-1586 Defaulting for the source type field in alerts UI does
not work (merrimanr via justinleet) closes apache/metron#1038
METRON-1569: Allow user to change field name conversion when
indexing to Elasticsearch (nickwallen via mmiklavc) closes
apache/metron#1022
METRON-1544 Flaky test:
org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
(nickwallen) closes apache/metron#1015
METRON-1580 Release candidate check script requires Bro Plugin
(nickwallen via ottobackwards) closes apache/metron#1034
METRON-1532 Getting started documentation improvements (sardell
via nickwallen) closes apache/metron#1001
METRON-1576 bundle.css RAT failure for
metron-interface/metron-alerts (justinleet) closes apache/metron#1029
METRON-1575 Add leet gpg public key to the KEYS file (justinleet)
closes apache/metron#1028
METRON-1574 Update version to 0.5.0 (justinleet) closes apache/metron#1026
METRON-1566 Alert updates are not propagated to metaalert child
alerts (merrimanr) closes apache/metron#1018
METRON-1565 Metaalerts fix denormalization after moving to active
status (merrimanr) closes apache/metron#1017
METRON-1548 Remove hardcoded source:type from Alerts UI
(justinleet) closes apache/metron#1010
METRON-1548 Remove hardcoded source:type from Alerts UI (sardell
via justinleet) closes apache/metron#1010
METRON-1564 Full dev kafka has offsets.topic.replication.factor
set to 3 instead of 1 (justinleet) closes apache/metron#1016
METRON-1552: Add gzip file validation check to the geo loader
(mmiklavc via mmiklavc) closes apache/metron#1011
METRON-1551 Profiler Should Not Use Java Serialization
(nickwallen) closes apache/metron#1012
METRON-1549: Add empty object test to WriterBoltIntegrationTest
implementation (mmiklavc via mmiklavc) closes apache/metron#1009
METRON-1541 Mvn clean results in git status having deleted files.
(justinleet via nickwallen) closes apache/metron#1003
METRON-1461 MIN MAX stellar function should take a stats or list
object and return min/max (MohanDV via nickwallen) closes
apache/metron#942
METRON-1184 EC2 Deployment - Updating control_path to accommodate
for Linux (Ahmed Shah via ottobackwards) closes apache/metron#754
METRON-1530 Default proxy config settings in metron-contrib need
to be updated (sardell via merrimanr) closes apache/metron#998
METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
apache/metron#1008
METRON-1543 Unable to Set Parser Output Topic in Sensor Config
(nickwallen) closes apache/metron#1007
METRON-1539: Specialized RENAME field transformer closes
apache/incubator-metron#1002
METRON-1520: Add caching for stellar field transformations closes
apache/incubator-metron#990
METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When Run in
Zeppelin REPL (nickwallen) closes apache/metron#997
METRON-1511 Unable to Serialize Profiler Configuration
(nickwallen) closes apache/metron#982
METRON-1528: Fix missing file in metron.spec (mmiklavc via
mmiklavc) closes apache/metron#996
METRON-1445: Update performance tuning guide with more explicit
parameter instructions (mmiklavc via mmiklavc) closes
apache/metron#988
METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
apache/metron#974
METRON-1527: Remove dead test file sitting in source folder
(mmiklavc via mmiklavc) closes apache/metron#994
METRON-1499 Enable Configuration of Unified Enrichment Topology
via Ambari (nickwallen) closes apache/metron#984
METRON-1515: Errors loading stellar functions currently bomb the
entire topology, they should be recoverable closes
apache/incubator-metron#985
METRON-1522 Fix the typo errors at profile debugger readme
(MohanDV via nickwallen) closes apache/metron#992
METRON-1519 Indexing Error Topic Property Not Displayed in MPack
(nickwallen) closes 

Solr Feature Branch

[Justin Leet]

Hi all,

An earlier thread on the dev list

discussed upgrading Solr and bringing it to feature parity with
Elasticsearch. We also wanted to inform the larger user list for anyone
who's interested in watching (or contributing!)

A Jira to track that effort is at
https://issues.apache.org/jira/browse/METRON-1416, along with an initial
set of subtasks towards this goal.

A feature branch has been created for this effort to make life easier for
everyone: feature/METRON-1416-upgrade-solr

Re: Enable geo enrichment

[Justin Leet]

There is also a Stellar function for doing geo lookups.
http://metron.apache.org/current-book/metron-stellar/stellar-common/index.html#GEO_GET
It'll
return a map of the fields when given an IP.

On Thu, Oct 5, 2017 at 5:37 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> And incase your install didn’t pick up the latest geo database (or you
> want to update it, the bottom of http://metron.apache.org/
> current-book/metron-platform/metron-data-management/index.html gives you
> the relevant info.
>
>
> On 5 Oct 2017, at 22:36, Simon Elliston Ball 
> wrote:
>
> http://metron.apache.org/current-book/metron-platform/
> metron-enrichment/index.html
>
> Shows you how to configure geo enrichment.
>
> Simon
>
> On 5 Oct 2017, at 22:33, Laurens Vets  wrote:
>
> What's the quickest way to enable geo enrichment on a source ip address in
> 0.4.1-release? Is there a simple document somewhere with instructions?
>
>
>
>

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

[Justin Leet]

Could we put these up on the wiki page for tech talks in the community?
That page could probably use some love, although I know we've had
discussions about what we should do with wiki content.

https://cwiki.apache.org/confluence/display/METRON/Tech+Talks

On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella  wrote:

> The Videos of talks that Simon Ball and I gave at DataWorks Summit are now
> up and on youtube:
>
> * Solving Cyber at Scale (business-level track) - https://www.youtube.com/
> watch?v=zVdRhwfum4Q
> * Model as a Service (technical track) - https://www.youtube.com/
> watch?v=LkrOKvyAc0s
> * Metron Architecture (with demo from LANL data) (technical track) -
> https://www.youtube.com/watch?v=0LrrAQXhqGY
>
> These talks are mostly current based on the existing architecture and the
> demos reflect the alerting UI that is not committed yet.  There are blogs
> coming out in support of this over the next week or so.
>
> If anyone has any questions about the talks or want any more information,
> feel free to ask. :)
>
> Best,
>
> Casey
>

Re: Geo enrichment failure after blocking internet connectivity

[Justin Leet]

My expectation is that /apps/metron/geo is empty (or at least has no files
in subdirs), can you verify this?

Assuming it is empty, you should be able to place the file (
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz)
into HDFS at /apps/metron/geo/default/GeoLite2-City.mmdb.gz and restart
enrichment.  It'll look there by default.

If you set up a cluster in the future, and want to avoid it Ambari also has
a config for the GeoIP file's URL, and I believe (but haven't looked at it
in awhile) that it should be able to take a file:/// type url that points
to a local file on the Metron master node, as long as that file exists
prior to Ambari's attempt to use it.

Let me know if that solves the problem; I haven't taken a look at that
stuff in a little bit, so I may have to dig a bit deeper if that doesn't
resolve it.

On Fri, Jul 28, 2017 at 1:13 AM, Ali Nazemian  wrote:

> Hi,
>
> Recently we have blocked internet connection to one of our platforms.
> After we had restarted Enrichment topology, we found out that topology
> cannot start anymore and it keeps throwing the following exception.
>
> 2017-07-28 04:41:38.816 o.a.c.f.r.c.TreeCache [ERROR]
>
> java.lang.IllegalStateException: [Metron] Unable to update MaxMind
> database
>
>at org.apache.metron.enrichment.
> adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:107)
> ~[stormjar.jar:?]
>
>at org.apache.metron.enrichment.
> adapters.geo.GeoLiteDatabase.updateIfNecessary(GeoLiteDatabase.java:71)
> ~[stormjar.jar:?]
>
>at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.
> reloadCallback(ThreatIntelJoinBolt.java:205) ~[stormjar.jar:?]
>
>at org.apache.metron.common.bolt.ConfiguredEnrichmentBolt.
> updateConfig(ConfiguredEnrichmentBolt.java:61) ~[stormjar.jar:?]
>
>at org.apache.metron.common.bolt.
> ConfiguredBolt$1.childEvent(ConfiguredBolt.java:91) ~[stormjar.jar:?]
>
>at 
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
> [stormjar.jar:?]
>
>at 
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
> [stormjar.jar:?]
>
>at 
> org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
> [stormjar.jar:?]
>
>at org.apache.metron.guava.util.concurrent.MoreExecutors$
> SameThreadExecutorService.execute(MoreExecutors.java:297) [stormjar.jar:?]
>
>at org.apache.curator.framework.listen.ListenerContainer.
> forEach(ListenerContainer.java:84) [stormjar.jar:?]
>
>at org.apache.curator.framework.recipes.cache.TreeCache.
> callListeners(TreeCache.java:678) [stormjar.jar:?]
>
>at 
> org.apache.curator.framework.recipes.cache.TreeCache.access$1400(TreeCache.java:69)
> [stormjar.jar:?]
>
>at org.apache.curator.framework.
> recipes.cache.TreeCache$4.run(TreeCache.java:790) [stormjar.jar:?]
>
>at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_131]
>
>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_131]
>
>at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_131]
>
>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_131]
>
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_131]
>
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_131]
>
>at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
>
>
> It seems there is a hard requirement for updating GeoEnrichment database
> that is broken now by blocking internet connection. How can we update that
> database manually and bypass the verification part of Metron for updating
> this database manually?
>
>
> Regards,
>
> Ali
>