Re: Metron capability to enrich a list granularity

[Otto Fowler]

Is there a Jira for the MAP Casey?


On April 6, 2017 at 14:07:15, Casey Stella (ceste...@gmail.com) wrote:

Ok, so yeah, you've hit upon a limitation currently.  Right now, via
Stellar you can use ENRICHMENT_GET which takes the following parameters:

   - enrichment_type - The enrichment type
   - indicator - The string indicator to look up
   - hbase_table - The HBase Table to use
   - column_family - The Column Family to use

Right now we only accept a string for the indicator (which likely would be
your user_id).  You'd probably like to call ENRICHMENT_GET for each id in
the user_id variable.  We can't quite do that yet.  There has been some
talk about a MAP function created where you can apply a stellar function
across a list of values.  i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
'enrichments', 't')) which would return a list containing the output of
ENRICHMENT_GET for each call.

There is another, more immediate change that could be made for this
specific case.  We could enable ENRICHMENT_GET to take a list of indicators
as the second argument.

Sorry, that doesn't exactly solve your problem in the immediate-case, but
it provides some context for future fixes. ;)  I don't suppose you know the
length of the list beforehand, right?  Even the maximum size?

Casey


On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian  wrote:

> Hi all,
>
>
> I was wondering how I can achieve the following use case in the current
> version of Metron?
>
>
>
> I want to have attributes in the Metron JSON object that are an array.
> For example, if a threat is impacting multiple users, they are all
> contained in an attribute (e.g.  user_id:[id1, id2, id3]).   Now if I want
> to enrich the event with data that requires the user_id as a key in
> enrichment stored in HBASE, how would I do this?
>
>
> Cheers,
> Ali
>

Re: integration.wrapper class does not exist

[Otto Fowler]

Are you running everything from the command line or from the IDE?
Can you just try from the command line?  Like > mvn integration-test


On March 31, 2017 at 01:22:59, moshe jarusalem (tuu...@gmail.com) wrote:

Hi all,
I have realized the functionality implemented in .scala files  should I do
anything to make java see those functions and classes ?





On Thu, Mar 30, 2017 at 7:18 PM, moshe jarusalem  wrote:

> Hi,
>
> metron-deployment/scripts/platform-info.sh
> Metron 0.3.1
> --
> * (HEAD detached at apache-metron-0.3.1-rc5-incubating)
> --
> commit 7abd7e8a231c6cbe9ee4ab23a5df1e97344f5212
> Author: justinleet 
> Date:   Thu Feb 23 10:40:14 2017 -0500
>
> METRON-734 Builds failing because of MaxMind DB transitive dependency
> (justinleet via cestella) closes apache/incubator-metron#462
> --
> --
> ansible 2.0.0.2
>   config file = /etc/ansible/ansible.cfg
>   configured module search path = Default w/o overrides
> --
> Vagrant 1.9.1
> --
> Python 2.7.12
> --
> Apache Maven 3.3.9
> Maven home: /usr/share/maven
> Java version: 1.8.0_121, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-66-generic", arch: "amd64", family:
> "unix"
> --
> Linux ubuntu 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> --
> Total System Memory = 11616.3 MB
> Processor Model: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
> Processor Speed: 2294.709 MHz
> Total Physical Processors: 4
> Total cores: 4
> Disk information:
> /dev/sda139G   18G   20G  47% /
>
>
>
> On Thu, Mar 30, 2017 at 5:54 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Hi Moshe, can you run the following script and post your results?
>>
>> metron-deployment/scripts/platform-info.sh
>>
>> Best,
>> Michael Miklavcic
>>
>>
>> On Thu, Mar 30, 2017 at 5:21 AM, moshe jarusalem 
>> wrote:
>>
>>>
>>> -- Forwarded message --
>>> From: moshe jarusalem 
>>> Date: Thu, Mar 30, 2017 at 2:20 PM
>>> Subject: integration.wrapper class does not exist
>>> To: d...@metron.incubator.apache.org
>>>
>>>
>>> Hi All,
>>> I am getting the following error while trying to run SquidParserTest.
>>> Would you help to resolve it?
>>>
>>>
>>> [image: Inline image 1]
>>>
>>>
>>
>


ii_15b1ef0abc9662ec
Description: Binary data

Re: snort topology doesn't emitted automatically

[Otto Fowler]

/opt/snort-producer/start-snort-producer.sh



On March 22, 2017 at 13:30:36, tkg_cangkul (yuza.ras...@gmail.com) wrote:

start_snort_producer.sh

Re: snort topology doesn't emitted automatically

[Otto Fowler]

One time, I saw an issue where the flume agent did not have the correct
rights
to access the csv, so died a horrible death.

We don’t use flume any longer however.  I would want to take a look at the
log files
for what is reading the snort csv.

I believe the start_snort_producer.sh script is used now.  I am not sure
about the logs,
but maybe you can try to run that manually and see the output?


On March 22, 2017 at 11:38:53, tkg_cangkul (yuza.ras...@gmail.com) wrote:

anyone can help me to solved this?

On 22/03/17 15:24, tkg_cangkul wrote:

hi, i've try to using snort as a sensor on metron in my ambari cluster.
now i've a problem. the snort topology doesn't emitted the data
automatically.
i must send the messages to kafka manually to emitted the data.

*cat /var/log/snort/alert.csv | bin/kafka-console-producer.sh --broker-list
localhost:6667 --topic snort*

any suggest about this?

Re: [ANNOUNCE] Apache Metron (incubating) 0.3.1 is released

[Otto Fowler]

That write up is more….. polished.  Worthy of your new position I think!

Congratulations everyone!


On March 17, 2017 at 11:53:25, Casey Stella (ceste...@gmail.com) wrote:

I am very proud to announce that the 0.3.1 release bits have been
released.  You can see this reflected on our website at
http://metron.apache.org/documentation/#releases  Also, I want to point out
that our github documentation for the release is currently located at
http://metron.apache.org/current-book/index.html and linked from the
release page (Thanks Matt for making that happen!).

I'm particularly proud of this release as it'll be the release on which we
base our exit from the incubator.  I really appreciate all of the
contributions that everyone made to make this possible.  Heartfelt
gratitude goes out to the community, the committers, the contributors and
the mentors for making this happen.  In the best tradition of open source
software, it took a village to build a Metron. :)

Highlights from this release:

   - Proper numeric types for Stellar
   - A CEF parser similar to the one in NiFi
   - HLLP+ sketches for Stellar and Profiler. Now you can answer questions
   like "# of distinct IPs did this user connect to?" in triage rules
   - The github documentation in a docbook
   - Transition of geo enrichment to not rely on mysql and have an
   accompanying Stellar function.
   - Enrichment Loader got faster and more capable.  Now you can do stellar
   transformations on data you're loading into HBase!
   - Stability and robustness improvements to the Profiler and the core
   Stellar functions.
   - Indexes can be turned on and off at the sensor granularity (i.e. write
   to ES without writing to HDFS)
   - Zeppelin notebooks!

Tweet linked at https://twitter.com/ApacheMetron/status/842761543620079616

Best,

Casey

PS. I still have some JIRA work to do to clean up from this release; I'll
be doing that by the end of the weekend.

Re: deployed metron on ambari

[Otto Fowler]

You created it in HDFS?


On February 23, 2017 at 21:53:49, tkg_cangkul (yuza.ras...@gmail.com) wrote:

i've created that dir before and i set the permission to 755.
is there any configuration that must be created before except create the
directory?

On 24/02/17 09:49, Casey Stella wrote:

Can you ensure that /apps/metron/indexing exists and is able to be written
from the user that Storm is running as?  If it's not, then you will need to
create the directory.

On Thu, Feb 23, 2017 at 6:47 PM, tkg_cangkul  wrote:

> hi ,
>
> i've tried to deploy metron on ambari. when i try to run indexing
> topology, i've found an error message like this :
>
>
> any suggestion about this? i'm using metron 0.3.0
>


part2.04080004.07030005@gmail
Description: Binary data