Re: [DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

2019-05-10 Thread Ali Nazemian
have nearly completed > decoupling our core infrastructure from Storm at this point, which opens us > up to a number of possibilities going forward. > > Best, > Mike Miklavcic > > > On Thu, Apr 4, 2019 at 1:35 AM Ali Nazemian wrote: > > > Hi All, > > > >

[DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

2019-04-04 Thread Ali Nazemian
Hi All, As far as I understood, there is a plan to change the real-time engine of Metron due to some issues that user and developer have been facing with it. I would like to explain some critical issues that customer have been facing to clarify it for the development team what the best approach

Re: [DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2019-01-07 Thread Ali Nazemian
Just one thing to bear in mind, publishing an error may cause some operational challenges as it fills up the error topic as well as storm logs which may not be necessary. To wear a Metron user hat, dropping a message with a debug/trace level log to specify the event is filter out makes sense. I

Re: [DISCUSS] Recurrent Large Indexing Error Messages

2018-12-05 Thread Ali Nazemian
I think if you look at the indexing error management, it is pretty much similar to parser and enrichment error use cases. It is even more common to expect something ended up in error topics. I think a wider independent job can be used to take care of error management. It can be decided to add a

Re: Authorization for Configuration

2018-12-03 Thread Ali Nazemian
n > > items as subtasks on the FB Jira so that we can crosscheck what entry > > points have been implemented against the test scripts. Do you think this > > will impact streaming enrichments or the profiler at all? That is to say, > > as Ali asked, just how far are you looking to t

Re: [DISCUSS] Deprecating MySQL

2018-11-19 Thread Ali Nazemian
Great feature to move to LDAP integration and hopefully Ranger integration afterwards. Does it need to support LDAP and AD separately? Cheers, Ali On Sat, Nov 17, 2018 at 3:29 AM Otto Fowler wrote: > I would like to understand the work required to move our JDBC support ( or > adapt the current

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

2018-11-19 Thread Ali Nazemian
Hi, One thing to point out here is there were a few timestamp fields that exist for Split-join enrichment topology that haven't been made to the unified one. For example, there is no threat intel bolt timestamp. There might be some SLA related use cases regarding these timestamp fields that might

Re: [DISCUSS] Authorization for Configuration

2018-11-19 Thread Ali Nazemian
Hi Justin, By configuration do you mean the sensor related configurations only? Are you limiting the scope of this activity to the management-UI or also Alert-UI as well? For example, defining different roles (pre-defined or customizable) and the fine-grained integration with Ranger? Cheers, Ali

Re: Authorization for Configuration

2018-11-19 Thread Ali Nazemian
Hi Justin, By configuration do you mean the sensor related configurations only? Are you limiting the scope of this activity to the management-UI or also Alert-UI as well? For example, defining different roles (pre-defined or customizable) and the fine-grained integration with Ranger? Cheers, Ali

Re: [DISCUSS] Slack Channel Use

2018-10-24 Thread Ali Nazemian
I kind of expect to have Slack for more dev related discussions rather than user QA. I guess it is quite common to expect mailing list to be used for the purpose of knowledge sharing to make sure it will be accessible by other users as well. Of course, it is a trade-off that most of the other

Re: HCP in Cloud infrastructures such as AWS , GCP, AZURE

2018-10-24 Thread Ali Nazemian
Depending on the model of security, you may have some challenges with the Ranger integration with your cloud storage especially if you are thinking of using TDE for the encryption at rest. Otherwise, using Metron in that way should be quite feasible. However, you may face some performance issues

Re: [DISCUSS] Internal Metron fields

2018-09-12 Thread Ali Nazemian
Totally agree with replacing dot with something else. We have had so much drama to use either dot or column with ORC either via Hive or Spark. Although we have replaced it with an underscore, it may not be a good idea as it can be confusing with underscores in the internal field names. Cheers,

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-27 Thread Ali Nazemian
Can I be invited as well? On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler wrote: > Done > > > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: > > Could I be invited? > > On 15-Aug-18 09:48, Michael Miklavcic wrote: > > + Metron user list > > > > On Wed, Aug 15, 2018 at

Re: [DISCUSS] Getting to a 1.0 release

2018-08-27 Thread Ali Nazemian
One thing that we could imagine for v1.0 might be an ability to extend Metron from adding more pipelines to it. For example, being able to extend Metron to be integrated with other endpoints more easily from Storm perspective. For example, what if we would like to create other topologies to write

Re: Change field separator in Metron to make it Hive and ORC friendly

2018-08-15 Thread Ali Nazemian
. > > Simon > > Sent from my iPhone > > > On 14 Aug 2018, at 11:42, deepak kumar wrote: > > > > I agree Ali. > > May be it can be configuration parameter. > > > >> On Tue, Aug 14, 2018 at 3:e t24 PM Ali Nazemian > wrote: > >> >

Re: Change field separator in Metron to make it Hive and ORC friendly

2018-08-14 Thread Ali Nazemian
; Do you have any suggestions for what would make sense as a delimiter? > > On 9 August 2018 at 05:57, Ali Nazemian wrote: > > > Hi All, > > > > I was wondering if we can change the field separators in Metron to be > able > > to make it Hive/ORC friendly. I could

Change field separator in Metron to make it Hive and ORC friendly

2018-08-08 Thread Ali Nazemian
Hi All, I was wondering if we can change the field separators in Metron to be able to make it Hive/ORC friendly. I could find the following PR, but neither dot nor colon is very Hive and ORC friendly and they will cause some issues. Hence, I wanted to see if it is possible to change the field

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-07-02 Thread Ali Nazemian
n Thu, Jun 14, 2018 at 2:28 PM Ali Nazemian wrote: > Hi Michael and Casey, > > It looks like ES believe Java Rest Client is mature enough to be pushed to > different products at this stage. However, I haven't used it personally. I > will share the question regarding x-pack

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Ali Nazemian
Casey Stella wrote: > > > It was my understanding was that ES x-pack only supports the transport > > client (e.g. > > https://www.elastic.co/guide/en/x-pack/current/java-clients.html). I > > think > > that was a major reason why we chose to go that route. I mi

Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Ali Nazemian
Hi All, I have noticed that the recommendation from Elasticsearch team is changed to use Java Rest Client instead of Transport one. The rationale behind it looks convincing and it can also help Metron to be more decoupled from Elasticsearch roadmap, so Metron users can upgrade Elasticsearch with

Re: Streaming Machine Learning use case

2018-05-09 Thread Ali Nazemian
hat we'd be better off looking at algorithms in Spark for > things like frequent pattern mining, though there the FP growth algorithm > is of course primarily a batch implementation. > > Are there any SAMOA algorithms in particular that you think would be > relevant to Metron use cases? > &g

Streaming Machine Learning use case

2018-05-08 Thread Ali Nazemian
Hi all, I was wondering if someone has used Metron with any streaming ML framework such as SAMOA? I know that Metron provides Machine Learning separately via MAAS. However, it is hard to manage it from operational perspective especially if we want to have a pretty dynamic and evolving model.

Re: [DISCUSS] Generic Syslog Parsing capability for parsers

2018-03-26 Thread Ali Nazemian
Just adding more details regarding what different parts are: There are three stages here that need to be understood: 1- pre-parsing 2- chain of parsing (wrapping one type of message in another format) 3- post-parsing aka normalization Pre-parsing stage is where we need to specify what specific

Re: [DISCUSS] community view/roadmap of threat intel

2018-02-21 Thread Ali Nazemian
gt; > > today the default Metron schema seems to lack any similar concept? Do > we > > > have plans to address it? > > > > > > 3. Atemporal matching - Given the use of big data technologies it seems > > to > > > me Metron should be able to look into past

ES mpack to include more ES 5 stack properties

2018-02-18 Thread Ali Nazemian
Hi All, Is there any plan to include more ES 5+ specific properties to Metron mpack? For example, if we want to use dedicated nodes for Master Nodes, Data Nodes, Ingestion Nodes and ML Nodes and different configurations for them, how can we proceed? It may be out of the scope of the current

Re: [DISCUSS] community view/roadmap of threat intel

2018-02-15 Thread Ali Nazemian
gt; intel loader, or even through a direct to hbase streaming connector. > > Simon > > > On 14 Feb 2018, at 03:13, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > Hi All, > > > > I would like to understand Metron community view on Threat Intel > >

[DISCUSS] community view/roadmap of threat intel

2018-02-13 Thread Ali Nazemian
Hi All, I would like to understand Metron community view on Threat Intel aggregators as well as the roadmap of threat intelligence and threat hunting. There are some open source options available regarding threat intel aggregator such as Minemeld, Hippocampe, etc. Is there any plan to build that

Re: Disable Metron parser output writer entirely

2018-02-05 Thread Ali Nazemian
ses only. It’s a very non-stellar construct > (non-expression, no return, side-effect dependent…) Also, it creates a > producer for every call, so your are definitely not going to get > performance out of it. > > Simon > > > On 5 Feb 2018, at 06:32, Ali Nazemian

Re: Disable Metron parser output writer entirely

2018-02-04 Thread Ali Nazemian
What about the performance difference? On Fri, Feb 2, 2018 at 10:41 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > You cannot. > > > > On February 1, 2018 at 23:51:28, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Hi All, > > I am trying to investi

Disable Metron parser output writer entirely

2018-02-01 Thread Ali Nazemian
Hi All, I am trying to investigate whether we can disable a Metron parser output writer entirely and manage it via KAFKA_PUT Stellar function instead. First, is it possible via configuration? Second, will be any performance difference between normal Kafka writer and the Stellar version of it

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
o I think if you change it in stellar it should work. Have you tried and failed? On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote: Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Are you trying to change th

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
And I am trying to understand if I set a post-parser Stellar transformation to change the value of "source.type" will it impact enrichment routing or it will get overwritten by an internal method? On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian <alinazem...@gmail.com> wrote:

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based > on the indexing config named the same as source.type -> wherever the > indexer tells it to be. > > Simon > > > On 29 Jan 2018, at 11:53, Ali Nazemian <alinazem...@gmail.com> wrote: > >

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Yes, it is. > > Sent from my iPhone > > > On 29 Jan 2018, at 09:33, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > Hi All, > > > > I was wondering how the routing mechanism works in

Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
Hi All, I was wondering how the routing mechanism works in Metron currently. Can somebody please explain how Enrichment Storm topology understands a single event is related to which Metron feed? What about indexing? is that based on "source.type" field? Cheers, Ali

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-24 Thread Ali Nazemian
Hi All, I just wanted to say it would be great if we can be careful with these type of changes. From the development point of view, it is just a few lines of code which can provide multiple advantages, but for live large-scale Metron platforms, some of these changes might be really expensive to

Re: Metron Alert UI and zero-down time Elasticsearch re-index

2018-01-14 Thread Ali Nazemian
It would be great if we can have some help on this issue. Cheers, Ali On Sat, Jan 6, 2018 at 12:33 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi James, > > Due to changes in the field format, I want to create a new index with the > new format. Create an alias to refer t

Re: Metron Alert UI and zero-down time Elasticsearch re-index

2018-01-05 Thread Ali Nazemian
; > 01.01.2018, 22:30, "Ali Nazemian" <alinazem...@gmail.com>: > > Hi All, > > > > We are using an older version of Metron Alert-UI (Received in Oct 2017) > > which sends search queries to ES directly without using Metron Rest API. > We > >

Metron Alert UI and zero-down time Elasticsearch re-index

2018-01-01 Thread Ali Nazemian
Hi All, We are using an older version of Metron Alert-UI (Received in Oct 2017) which sends search queries to ES directly without using Metron Rest API. We wanted to run a zero-downtime ES reindex process by using ES aliasing. However, I am not sure how it will impact the search part of Alert-UI

Re: Metron nested object

2017-12-21 Thread Ali Nazemian
; > On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Hi all, > > > We have recently faced some data sources that generate data in a nested > format. For example, AWS Cloudtrail generates data in the following JSON > format: > >

Metron nested object

2017-12-21 Thread Ali Nazemian
Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*,

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-11 Thread Ali Nazemian
he levers outlined above, iterate with each change in rapid >succession, and record your results. > > 1. > https://github.com/apache/metron/blob/master/metron- > platform/Performance-tuning-guide.md > > Sample command without Kerberos enabled (see link [1] for more detail wit

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-10 Thread Ali Nazemian
d be wrong though. > > > > > On December 7, 2017 at 06:47:15, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Thank you very much. Unfortunately, reproducing all the situations are > very costly for us at this moment. We are kind of avoiding to hit that > issue by u

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-07 Thread Ali Nazemian
That code does not have any logging to speak of… well debug / trace > logging that would help here either. > > > > On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Everything looks normal except the high number of failed tuples. Do you > know how the

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-06 Thread Ali Nazemian
com> wrote: > What do you see in the storm ui for the indexing topology? > > > On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It > mpacts topology error rate a

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-06 Thread Ali Nazemian
er 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com) wrote: Which of the indexing options are you changing the batch size for? HDFS? Elasticsearch? Both? Can you give an example? On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote: No specific error in the

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-04 Thread Ali Nazemian
No specific error in the logs. I haven't enabled debug/trace, though. On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > My first thought is what are the errors when you get a high error rate? > > > On December 4, 2017 at 19:34:29, Ali Nazemian (ali

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-04 Thread Ali Nazemian
Any thoughts? On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi, > > We have noticed recently that no matter what batch size we use for Metron > indexing feeds, as long as we start using different batch size for > different Metron fee

Heterogeneous indexing batch size for different Metron feeds

2017-12-03 Thread Ali Nazemian
Hi, We have noticed recently that no matter what batch size we use for Metron indexing feeds, as long as we start using different batch size for different Metron feeds, indexing topology throughput will start dropping due to the high error rate! So I was wondering whether based on the current

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread Ali Nazemian
on, Nov 27, 2017 at 3:46 PM, James Sirota <jsir...@apache.org> wrote: > > > One thing to keep in mind, as we will be introducing Solr shortly, is to > > find if something similar to curator exists for Solr. But we'll cross > that > > bridge when we get there > &g

Re: Using Storm Resource Aware Scheduler

2017-11-26 Thread Ali Nazemian
is no need for the new JIRA ( > https://issues.apache.org/jira/browse/METRON-1330 < > https://issues.apache.org/jira/browse/METRON-1330>). It should be closed > as a duplicate of https://issues.apache.org/jira/browse/METRON-1161 < > https://issues.apache.org/jira/browse/

Re: Using Storm Resource Aware Scheduler

2017-11-23 Thread Ali Nazemian
Any help regarding this question would be appreciated. On Thu, Nov 23, 2017 at 8:57 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > 30 mins average of CPU load by checking Ambari. > > On 23 Nov. 2017 00:51, "Otto Fowler" <ottobackwa...@gmail.com> w

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-22 Thread Ali Nazemian
ke > > On Nov 22, 2017 8:53 PM, "Ali Nazemian" <alinazem...@gmail.com> wrote: > > > We tried to use it, but we had the same issue. It was not documented. We > > tried to use it, and we had some issues. It also was not exactly what we > > wanted, so we decid

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-22 Thread Ali Nazemian
We tried to use it, but we had the same issue. It was not documented. We tried to use it, and we had some issues. It also was not exactly what we wanted, so we decided to create something from scratch by using Elasticsearch Curator. We wanted to have an ability to manage different prune mechanism

Re: Using Storm Resource Aware Scheduler

2017-11-22 Thread Ali Nazemian
30 mins average of CPU load by checking Ambari. On 23 Nov. 2017 00:51, "Otto Fowler" <ottobackwa...@gmail.com> wrote: How are you measuring the utilization? On November 22, 2017 at 08:12:51, Ali Nazemian (alinazem...@gmail.com) wrote: Hi all, One of the issues tha

Using Storm Resource Aware Scheduler

2017-11-22 Thread Ali Nazemian
Hi all, One of the issues that we are dealing with is the fact that not all of the Metron feeds have the same type of resource requirements. For example, we have some feeds that even a single Strom slot is way more than what it needs. We thought we could make it more utilised in total by

Re: Metron 0.4.2 release date

2017-10-11 Thread Ali Nazemian
l.com>: > > > There's an ongoing conversation regarding client support in Metron here > > > <https://lists.apache.org/thread.html/0c5a837c901dd057420dd8c6b673dc > > 33ba88a8d97545d5b58856cfe8@%3Cdev.metron.apache.org%3E> > > > . > > >

Re: Metron 0.4.2 release date

2017-10-08 Thread Ali Nazemian
hs, and trying to look further into the future than that at this > point would be difficult. > > That said, if anybody else has a more detailed timeline in mind, I would > love to hear more. > > Jon > > On Sun, Oct 8, 2017, 09:05 Ali Nazemian <alinazem...@gmail.com> wr

Metron 0.4.2 release date

2017-10-08 Thread Ali Nazemian
Hi all, I was wondering when Metron 0.4.2 will be released and whether it includes Metron-777 and Elasticsearch 5.x or not? Cheers, Ali

Elasticsearch 5.x upgrade

2017-07-16 Thread Ali Nazemian
Hi all, I've heard there is a plan to upgrade Elasticsearch from 2.x to 5.x regarding Metron and Ambari mpack. I was wondering when that will happen. Is there any part in Metron Elasticsearch indexing that will be impacted by this upgrade? Like any change from the way of bulk-indexing? Cheers,

Re: UI pivotting / aggregation backend

2017-07-08 Thread Ali Nazemian
Given the fact that some people prefer Solr and some of them Elasticsearch, having an abstraction layer for Solr and Elasticsearch would be really great. However, I haven't seen any framework out there that can provide the required level of search abstraction on top of Solr and Elasticsearch, but

Re: Post-parsing and Enrichment test framework

2017-07-08 Thread Ali Nazemian
. > > > > [Stellar]>>> > > [Stellar]>>> ip_src_addr := "10.0.0.2" > > [Stellar]>>> ip_dst_addr := "10.0.0.3" > > [Stellar]>>> ip_src_port := 22 > > [Stellar]>>> ip_dst_port := 12345 > > [Stel

Re: Post-parsing and Enrichment test framework

2017-07-04 Thread Ali Nazemian
track of changes. Cheers, Ali On Wed, Jul 5, 2017 at 12:06 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > You should probably use the Stellar REPL (../metron/bin/stellar -z $ZK) > which gives you a kind of Stellar playground. > > Simon > > > On 4 Jul 2

Post-parsing and Enrichment test framework

2017-07-04 Thread Ali Nazemian
Hi all, I was wondering if there is a test framework we can use for Stellar post-parsing and enrichment use cases. It is very time-consuming to verify use cases end-to-end. Therefore, I am looking for a way of mocking use cases step by step to speed up our development. Regards, Ali

Re: performance benchmarks on the asa parser

2017-06-09 Thread Ali Nazemian
Simon, I have read all emails and now I understand what you are saying. However, I couldn't understand the effect of predictability of latency on enrichments. On Fri, Jun 9, 2017 at 2:45 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi Simon, > > We have noticed those issu

Re: performance benchmarks on the asa parser

2017-06-08 Thread Ali Nazemian
Hi Simon, We have noticed those issues as well. Can you share the changes you have made? so we can merge it with our version. We have implemented about 40-50 more ciscotags so far. It would be great if we can optimize it and contribute back to the community. However, we may end up reimplement it

Re: [Discuss] Cyber Security Asset Management for Metron

2017-05-24 Thread Ali Nazemian
; > > > It might be good to discuss in the community specific use cases that > would > > be enabled by a graph database. That might help to flesh out the > technical > > aspects of it. > > > > > > > > > > > > On Wed, May 2

[Discuss] Cyber Security Asset Management for Metron

2017-05-24 Thread Ali Nazemian
Hi all, We are going to design and develop an asset database for Metron. For this purpose, I have been thinking of a graph schema model to map assets as Nodes and provide relations as Edges. This can be extended to event level to have a particular relation to assets as well as an event to event

Re: Normalization topology or separate normalization bolt for parsing topology

2017-05-02 Thread Ali Nazemian
arser that matches your definition of "minimum set". > > My main point here is that I am not seeing a need to re-architect > anything. I think we have the right tools, IMHO. > > > > > > > > > > On Tue, May 2, 2017 at 10:33 AM, Ali Nazemian <alinaze

Re: Normalization topology or separate normalization bolt for parsing topology

2017-05-02 Thread Ali Nazemian
tackle a problem like this. Not all data can be > trusted. > > > > > > > > On Thu, Apr 27, 2017 at 9:54 AM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Are you sure? The syslog_host name is way more complicated than something > > that can be a

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-27 Thread Ali Nazemian
n from > malformed logs, rather than throwing exceptions, but that's more about the > way we write parsers than having some kind of pre-clean. > > Simon > > Sent from my iPad > > > On 27 Apr 2017, at 08:04, Ali Nazemian <alinazem...@gmail.com> wrote: > &g

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-27 Thread Ali Nazemian
m, no? > > Simon > > > > > On 27 Apr 2017, at 02:08, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > Hi Simon, > > > > The reason I am asking for a specific normalisation step is due to the > fact > > that normalisation is not a general

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Ali Nazemian
t; > > > On 26 Apr 2017, at 14:37, Casey Stella <ceste...@gmail.com> wrote: > > > > > > Ok, that's another story. h, we don't generally pre-parse becuase > we > > > try to not assume any particular format there (i.e. it could be > strings, > > >

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Ali Nazemian
(best effort tyep of thing) through the parser and do > the normalization post-parse..or is there a problem with that? > > On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Hi Casey, > > > > It is actually pre-parse proce

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Ali Nazemian
Stella <ceste...@gmail.com> wrote: > So, further transformation post-parse was one of the motivating reasons for > Stellar (to do that transformation post-parse). Is there a capability that > it's lacking that we can add to fit your usecase? > > On Wed, Apr 26, 2017 at

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Ali Nazemian
I've created a Jira ticket regarding this feature. https://issues.apache.org/jira/browse/METRON-893 On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Currently, we are using normal regex at the Java source code to handle > those situations. Howev

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Ali Nazemian
, Otto Fowler <ottobackwa...@gmail.com> wrote: > Hi, > > Are you doing this cleansing all in the parser or are you using any > Stellar to do it? > Can you create a jira? > > > > On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com) wrote: > > Hi all

Re: So we graduated...

2017-04-21 Thread Ali Nazemian
That's great! Congratulation everybody. On Fri, Apr 21, 2017 at 12:54 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Congrats all > > On Apr 20, 2017 8:38 PM, "zeo...@gmail.com" wrote: > > > Well done everybody! Congrats > > > > Jon > > > > On Thu, Apr 20, 2017