Hi William,
 Was going through the ES watcher post our call and since we are using ES
2.4.1 as of now and will go over this x-pack alerting and let know.

Thanks again and kindly do let us know once profiling is available for us
to use.


Regards,
Ananthanarayanan.M.S

On Mon, Aug 7, 2017 at 2:33 PM, William Guo <gu...@apache.org> wrote:

>
> hi Ananthanarayanan,
>
>
> There are several ways to set up elasticsearch alerting.
>
> 1. alerts with x-pack
>
> x-pack is provided by Elastic Corporation. More information is availabe at
> https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html.
> x-pack requires license after evaluation period. The overall settings up
> are like the following.
>
>
>    - install x-pack plugin on all the nodes in the cluster
>    - set up channels like email server, slack server in the
>    elasticsearch.yml file
>    - set up alerts with desired query.
>    - when the desired query meet its condition, alerts will be issued via
>    channels.
>
>
>
>
> 2. free alternative
>
> ElastAlert is a simple framework for alerting on anomalies, spikes, or
> other patterns of interest from data in Elasticsearch. It's implemented in
> python. More information available at https://elastalert.
> readthedocs.io/en/latest/
>
> The overall settings up are like the following.
>
>    - install python packages elastalert and elasticsearch
>    - set up config.yaml for the connections and authentications
>    - set up the elasticsearch index for ElastAlert
>    - set up alerts
>    - run ElastAlert as a demon service or a python process
>    - when alerts meet their conditions, alerts will be issued via
>    channels.
>
>
> Thanks,
>
> William
>
>
>
> ------------------------------
> *From:* William GUO <guo...@outlook.com> on behalf of William Guo <
> gu...@apache.org>
> *Sent:* Friday, August 4, 2017 1:37:47 PM
> *To:* Ananthanarayanan Ms; Mara Preotescu
> *Cc:* dev@griffin.incubator.apache.org; Kunduru, Abishek
> *Subject:* Re: Meeting minutes with Nielsen:
>
> hi Ananthanarayanan,
>
>
> For profiling, we have developed some samples based on our measures. Will
> make samples available in our repo next week.
>
>
> Accuracy, we have test cases for it, but we will also make samples in our
> repo next week.
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Ananthanarayanan Ms <ananthanarayanan.ms...@nielsen.com>
> Sent: Friday, August 4, 2017 12:30:22 AM
> To: Mara Preotescu
> Cc: William Guo; dev@griffin.incubator.apache.org; Kunduru, Abishek
> Subject: Re: Meeting minutes with Nielsen:
>
> Hi William/Lionel,
>   Could you please help us to understand on the profiling feature
> availability, we see the code traces on profiling, could you let us know if
> we can starting using it from some branch which could be used so that we
> could leverage griffin on two dimensions (accuracy & profiling). If its not
> available a tentative date so that we could decide upon the same.
>
>
> Regards,
> Ananthanarayanan.M.S
>
> On Wed, Aug 2, 2017 at 10:47 PM, Mara Preotescu <
> mara.preote...@nielsen.com<mailto:mara.preote...@nielsen.com>> wrote:
> Thank you William.
>
> Ananth will follow up with a few more questions on the roadmap.
>
> Thanks,
> Mara
>
> On Wed, Aug 2, 2017 at 9:20 AM, William Guo <gu...@apache.org<mailto:guoyp
> @apache.org>> wrote:
>
>
> Meeting minutes with Nielsen:
>
>
>   *   Discuss griffin to support filters for metastore tables or
> navigation assistance for table selection on UI.
>   *   Griffin provides RESTful API for backend.
>   *   Discuss griffin to support multiple source or target tables.
>   *   Discuss more supporting file types, such as parquet.
>   *   In griffin, the partition field is optional, it just helps to
> provide the specific part of data, it will get all the data of a table
> without any partition information.
>   *   Config json file provides the parameters for griffin measure
> calculation, you can also submit a spark job with it directly.
>   *   Currently, griffin can only reuse measure, not rule. We’ll discuss
> about this, if we need to support reusing rules.
>   *   Sample ratio field in config file is optional, in batch mode we
> don’t need to configure it.
>   *   In griffin, mapping of columns are limited, discuss to support
> advanced features like joining between tables , or advanced sql script.
>   *   At current, the rule parser doesn’t support customized rules,
> griffin has the plan to support this. //TODO document it and send it to dev
> list
>   *   Griffin doesn’t support metrics alert function, it posts all the
> metrics to elasticsearch, es supports such feature. //TODO, write a
> solution for it based on elastic search
>   *   In griffin, you can’t modify the exist rules or measure at current.
>
>
>
> Thanks,
> William
>
>
> ________________________________
> From: William GUO <guo...@outlook.com<mailto:guo...@outlook.com>> on
> behalf of William Guo <gu...@apache.org<mailto:gu...@apache.org>>
> Sent: Wednesday, August 2, 2017 10:02:15 AM
> To: Mara Preotescu
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
> Subject: Re: Griffin support & roadmap
>
> hi mara,
>
>
> Are you join?
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preote...@nielsen.com<mailto:
> mara.preote...@nielsen.com>>
> Sent: Monday, July 31, 2017 11:22:00 PM
> To: William Guo
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org>; Ananthanarayanan Ms; Kunduru, Abishek
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Would 10:00 am CST (Beijing) work for you on Wednesday 08/02?
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 10:59 PM, William Guo <gu...@apache.org<mailto:
> gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>
> wrote:
>
> hi Mara,
>
>
> We are in China, it is hard to arrange a meeting for US, CHINA, INDIA
> together.
>
>
> China day time is fine for me.
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preote...@nielsen.com<mailto:
> mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.
> preote...@nielsen.com>>>
> Sent: Monday, July 31, 2017 10:54:25 AM
>
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org
> <mailto:dev@griffin.incubator.apache.org>>
> Subject: Re: Griffin support & roadmap
>
> Hi William,
>
> Either Wednesday or Thursday will work for us.  Any better time working
> for you?   What time zone are you in?   I am in US ET time, a colleague of
> mine who I would like to join our discussion is in India, Chennai.
>
> Thanks,
> Mara
>
> On Sun, Jul 30, 2017 at 7:34 PM, William Guo <gu...@apache.org<mailto:
> gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>
> wrote:
>
> hi Mara,
>
>
> Sure, We could schedule a meeting to discuss background, requirements,
> status and milestone.
>
>
> We should be fine in Wednesday or Thursday, what is your proposal?
>
>
>
> Thanks,
>
> William
>
> ________________________________
> From: Mara Preotescu <mara.preote...@nielsen.com<mailto:
> mara.preote...@nielsen.com><mailto:mara.preote...@nielsen.com<mailto:mara.
> preote...@nielsen.com>>>
> Sent: Friday, July 28, 2017 7:57:45 PM
> To: William Guo
> Cc: Lv, Alex; Guo, William; dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org><mailto:dev@griffin.incubator.apache.org
> <mailto:dev@griffin.incubator.apache.org>>
> Subject: Re: Griffin support & roadmap
>
> HI Alex, William,
>
> THANK YOU so much for your responses.  Thank you for the links.  And, I
> hope you don't mind if I'll take up your offer to contact you if needed.
> We are considering, here at Nielsen, using Griffin for our new Data Quality
> framework  ... we know the project is still in the incubator but we would
> like give it a try and even contributing, if needed.   We already install
> it and ran a few tests.
>
> If your time permits I would like scheduling a quick call so we could
> understand the current status and, most importantly if the roadmap stays as
> in the published documents.
>
> Thanks again,
> Mara
>
>
> On Fri, Jul 28, 2017 at 4:21 AM, William Guo <gu...@apache.org<mailto:
> gu...@apache.org><mailto:gu...@apache.org<mailto:gu...@apache.org>>>
> wrote:
>
> hi Mara,
>
> Few links might help, you can contact us by dev@griffin.incubator.apache.
> org<mailto:dev@griffin.incubator.apache.org><mailto:d
> e...@griffin.incubator.apache.org<mailto:dev@griffin.incubator.apache.org>>
> or my personal account gu...@apache.org<mailto:guoyp@
> apache.org><mailto:gu...@apache.org
> <gu...@apache.org%3E%3Cmailto:gu...@apache.org>>
>
>
> GitHub : https://github.com/apache/incubator-griffin<https://
> github.com/eBay/griffin<https://github.com/apache/incubator-
> griffin%3Chttps://github.com/eBay/griffin
> <https://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin%3Chttps://github.com/apache/incubator-griffin%3Chttps://github.com/eBay/griffin>
> >>
> Website : https://griffin.incubator.apache.org<https://griffin.
> incubator.apache.org/>
> Contact: mailto://subscribe-...@griffin.incubator.apache.org<
> mailto:subscribe-...@griffin.incubator.apache.org>
> Apache Griffin JIRA: https://issues.apache.org/jira/browse/GRIFFIN
> Apache Griffin Wiki :https://cwiki.apache.org/confluence/display/GRIFFIN/
> Griffin
>
> Thanks, William
>
> ________________________________
> From: Lv, Alex <lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:
> lzhix...@ebay.com<mailto:lzhix...@ebay.com>>>
> Sent: Friday, July 28, 2017 9:10:09 AM
> To: Mara Preotescu; Guo, William; gu...@apache.org<mailto:guoyp@
> apache.org><mailto:gu...@apache.org
> <gu...@apache.org%3E%3Cmailto:gu...@apache.org>>
> Cc: dev@griffin.incubator.apache.org<mailto:dev@griffin.
> incubator.apache.org><mailto:dev@griffin.incubator.apache.org<mailto:
> dev@griffin.incubator.apache.org>>
> Subject: RE: Griffin support & roadmap
>
> <<Move Amber to BCC>>
> Hi Mara,
>
> Glad to hear from you, you may discuss the details with William.
> Thx.
>
> Best regards,
> Alex Lv
>
> From: Mara Preotescu [mailto:mara.preote...@nielsen.com<mailto:mara.
> preote...@nielsen.com<mailto:mara.preote...@nielsen.com%
> 3Cmailto:mara.preote...@nielsen.com
> <mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com%3Cmailto:mara.preote...@nielsen.com>
> >>]
> Sent: 2017年7月28日 6:14
> To: Lv, Alex <lzhix...@ebay.com<mailto:lzhix...@ebay.com><mailto:lzhixing
> @ebay.com<mailto:lzhix...@ebay.com>>>; Vaidya, Amber <amvai...@ebay.com
> <mailto:amvai...@ebay.com><mailto:amvai...@ebay.com<mailto:amvaidya@
> ebay.com>>>
> Subject: Griffin support & roadmap
>
> Hello Alex, Amber,
>
> I am writing you trying to reach the support for Griffin, both support
> e-mails for the product returned as invalid addresses (
> subscribe-...@griffin.incubator.apache.org<mailto:su
> bscribe-...@griffin.incubator.apache.org><mailto:subscribe-
> dev@griffin.incubator.apache.org<mailto:subscribe-dev@
> griffin.incubator.apache.org>>,   ebay-griffin-d...@googlegroups.com
> <mailto:ebay-griffin-d...@googlegroups.com><mailto:ebay-griffin-devs@
> googlegroups.com<mailto:ebay-griffin-d...@googlegroups.com>>).
>
> Could you please let me know who should we contact to discuss about
> Griffin's roadmap?
>
> We are looking, here at Nielsen, to use the Griffin framework for our DQ
> processes.  As of today we learned,  and tested, the only dimension
> available, Accuracy.   Would you be able to share the roadmap for any other
> DQ dimensions availability?
>
> We are looking as well to add a few custom validations - does the tool
> offer any APIs that can be used for this purpose?
>
> Any information you could provide would be very, very helpful.
>
>
> Thank you in advance for your help and time.
> Mara Preotescu
> VP Technology,  DevOps Nielsen
>
>
>
>
>
>

Reply via email to