Sure, sample for profiling is on our agenda.


________________________________
From: Ananthanarayanan Ms <ananthanarayanan.ms...@nielsen.com>
Sent: Monday, August 7, 2017 5:13:51 PM
To: William Guo
Cc: Mara Preotescu; dev@griffin.incubator.apache.org; Feng Pan; Kunduru, Abishek
Subject: Re: Setup ES alerting //Re: Meeting minutes with Nielsen:

Hi William,
 Was going through the ES watcher post our call and since we are using ES 2.4.1 
as of now and will go over this x-pack alerting and let know.

Thanks again and kindly do let us know once profiling is available for us to 
use.


Regards,
Ananthanarayanan.M.S

On Mon, Aug 7, 2017 at 2:33 PM, William Guo 
<gu...@apache.org<mailto:gu...@apache.org>> wrote:


hi Ananthanarayanan,


There are several ways to set up elasticsearch alerting.

1. alerts with x-pack

x-pack is provided by Elastic Corporation. More information is availabe at 
https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html. x-pack 
requires license after evaluation period. The overall settings up are like the 
following.


  *   install x-pack plugin on all the nodes in the cluster
  *   set up channels like email server, slack server in the elasticsearch.yml 
file
  *   set up alerts with desired query.
  *   when the desired query meet its condition, alerts will be issued via 
channels.



2. free alternative

ElastAlert is a simple framework for alerting on anomalies, spikes, or other 
patterns of interest from data in Elasticsearch. It's implemented in python. 
More information available at https://elastalert.readthedocs.io/en/latest/

The overall settings up are like the following.

  *   install python packages elastalert and elasticsearch
  *   set up config.yaml for the connections and authentications
  *   set up the elasticsearch index for ElastAlert
  *   set up alerts
  *   run ElastAlert as a demon service or a python process
  *   when alerts meet their conditions, alerts will be issued via channels.


Thanks,

William



________________________________
From: William GUO <guo...@outlook.com<mailto:guo...@outlook.com>> on behalf of 
William Guo <gu...@apache.org<mailto:gu...@apache.org>>
Sent: Friday, August 4, 2017 1:37:47 PM
To: Ananthanarayanan Ms; Mara Preotescu
Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>; 
Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:

hi Ananthanarayanan,


For profiling, we have developed some samples based on our measures. Will make 
samples available in our repo next week.


Accuracy, we have test cases for it, but we will also make samples in our repo 
next week.


Thanks,

William

________________________________
From: Ananthanarayanan Ms 
<ananthanarayanan.ms...@nielsen.com<mailto:ananthanarayanan.ms...@nielsen.com>>
Sent: Friday, August 4, 2017 12:30:22 AM
To: Mara Preotescu
Cc: William Guo; 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>; 
Kunduru, Abishek
Subject: Re: Meeting minutes with Nielsen:

Hi William/Lionel,
  Could you please help us to understand on the profiling feature availability, 
we see the code traces on profiling, could you let us know if we can starting 
using it from some branch which could be used so that we could leverage griffin 
on two dimensions (accuracy & profiling). If its not available a tentative date 
so that we could decide upon the same.


Regards,
Ananthanarayanan.M.S

On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu 
<mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>>>
 wrote:
Thank you William.

Ananth will follow up with a few more questions on the roadmap.

Thanks,
Mara

On Wed, Aug 2, 2017 at 9:20 AM, William Guo 
<gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>
 wrote:


Meeting minutes with Nielsen:


  *   Discuss griffin to support filters for metastore tables or navigation 
assistance for table selection on UI.
  *   Griffin provides RESTful API for backend.
  *   Discuss griffin to support multiple source or target tables.
  *   Discuss more supporting file types, such as parquet.
  *   In griffin, the partition field is optional, it just helps to provide the 
specific part of data, it will get all the data of a table without any 
partition information.
  *   Config json file provides the parameters for griffin measure calculation, 
you can also submit a spark job with it directly.
  *   Currently, griffin can only reuse measure, not rule. We’ll discuss about 
this, if we need to support reusing rules.
  *   Sample ratio field in config file is optional, in batch mode we don’t 
need to configure it.
  *   In griffin, mapping of columns are limited, discuss to support advanced 
features like joining between tables , or advanced sql script.
  *   At current, the rule parser doesn’t support customized rules, griffin has 
the plan to support this. //TODO document it and send it to dev list
  *   Griffin doesn’t support metrics alert function, it posts all the metrics 
to elasticsearch, es supports such feature. //TODO, write a solution for it 
based on elastic search
  *   In griffin, you can’t modify the exist rules or measure at current.



Thanks,
William


________________________________
From: William GUO 
<guo...@outlook.com<mailto:guo...@outlook.com><mailto:guo...@outlook.com<mailto:guo...@outlook.com>>>
 on behalf of William Guo 
<gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>
Sent: Wednesday, August 2, 2017 10:02:15 AM
To: Mara Preotescu
Cc: 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>;
 Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap

hi mara,


Are you join?


Thanks,

William

________________________________
From: Mara Preotescu 
<mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>>>
Sent: Monday, July 31, 2017 11:22:00 PM
To: William Guo
Cc: 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>;
 Ananthanarayanan Ms; Kunduru, Abishek
Subject: Re: Griffin support & roadmap

Hi William,

Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?

Thanks,
Mara

On Sun, Jul 30, 2017 at 10:59 PM, William Guo 
<gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>><mailto:gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>>
 wrote:

hi Mara,


We are in China, it is hard to arrange a meeting for US, CHINA, INDIA together.


China day time is fine for me.


Thanks,

William

________________________________
From: Mara Preotescu 
<mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>>>>
Sent: Monday, July 31, 2017 10:54:25 AM

To: William Guo
Cc: Lv, Alex; Guo, William; 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>>
Subject: Re: Griffin support & roadmap

Hi William,

Either Wednesday or Thursday will work for us.  Any better time working for 
you?   What time zone are you in?   I am in US ET time, a colleague of mine who 
I would like to join our discussion is in India, Chennai.

Thanks,
Mara

On Sun, Jul 30, 2017 at 7:34 PM, William Guo 
<gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>><mailto:gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>>
 wrote:

hi Mara,


Sure, We could schedule a meeting to discuss background, requirements, status 
and milestone.


We should be fine in Wednesday or Thursday, what is your proposal?



Thanks,

William

________________________________
From: Mara Preotescu 
<mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>>>>
Sent: Friday, July 28, 2017 7:57:45 PM
To: William Guo
Cc: Lv, Alex; Guo, William; 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>>
Subject: Re: Griffin support & roadmap

HI Alex, William,

THANK YOU so much for your responses.  Thank you for the links.  And, I hope 
you don't mind if I'll take up your offer to contact you if needed.   We are 
considering, here at Nielsen, using Griffin for our new Data Quality framework  
... we know the project is still in the incubator but we would like give it a 
try and even contributing, if needed.   We already install it and ran a few 
tests.

If your time permits I would like scheduling a quick call so we could 
understand the current status and, most importantly if the roadmap stays as in 
the published documents.

Thanks again,
Mara


On Fri, Jul 28, 2017 at 4:21 AM, William Guo 
<gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>><mailto:gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>>
 wrote:

hi Mara,

Few links might help, you can contact us by 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>>
 or my personal account 
gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org%3E%3Cmailto:gu...@apache.org>>


GitHub : 
https://github.com/apache/incubator-griffin<https://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin<https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin%3Chttps://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>>>
Website : 
https://griffin.incubator.apache.org<https://griffin.incubator.apache.org/>
Contact: 
mailto://subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org>
Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/Griffin

Thanks, William

________________________________
From: Lv, Alex 
<lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com>><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com>>>>
Sent: Friday, July 28, 2017 9:10:09 AM
To: Mara Preotescu; Guo, William; 
gu...@apache.org<mailto:gu...@apache.org><mailto:gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org%3E%3Cmailto:gu...@apache.org>>
Cc: 
dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>>
Subject: RE: Griffin support & roadmap

<<Move Amber to BCC>>
Hi Mara,

Glad to hear from you, you may discuss the details with William.
Thx.

Best regards,
Alex Lv

From: Mara Preotescu 
[mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com>>>]
Sent: 2017年7月28日 6:14
To: Lv, Alex 
<lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com>><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:lzhix...@ebay.com<mailto:lzhix...@ebay.com>>>>;
 Vaidya, Amber 
<amvai...@ebay.com<mailto:amvai...@ebay.com><mailto:amvai...@ebay.com<mailto:amvai...@ebay.com>><mailto:amvai...@ebay.com<mailto:amvai...@ebay.com><mailto:amvai...@ebay.com<mailto:amvai...@ebay.com>>>>
Subject: Griffin support & roadmap

Hello Alex, Amber,

I am writing you trying to reach the support for Griffin, both support e-mails 
for the product returned as invalid addresses 
(subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org><mailto:subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org>><mailto:subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org><mailto:subscribe-...@griffin.incubator.apache.org<mailto:subscribe-...@griffin.incubator.apache.org>>>,
   
ebay-griffin-d...@googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com><mailto:ebay-griffin-d...@googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com>><mailto:ebay-griffin-d...@googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com><mailto:ebay-griffin-d...@googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com>>>).

Could you please let me know who should we contact to discuss about Griffin's 
roadmap?

We are looking, here at Nielsen, to use the Griffin framework for our DQ 
processes.  As of today we learned,  and tested, the only dimension available, 
Accuracy.   Would you be able to share the roadmap for any other DQ dimensions 
availability?

We are looking as well to add a few custom validations - does the tool offer 
any APIs that can be used for this purpose?

Any information you could provide would be very, very helpful.


Thank you in advance for your help and time.
Mara Preotescu
VP Technology,  DevOps Nielsen






Reply via email to