Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
Agree that it is the just time moving Apache Griffin forward, to be a TLP. On Tue, Oct 16, 2018 at 5:46 PM jenny li wrote: > Agree with William's proposal. > > Best Regards, > Jenny > > On Tue, Oct 16, 2018 at 5:33 PM Lionel Liu wrote: > > > As I've commented in dev list, I agree with this proposal. > > > > Thanks, > > Lionel > > > > On Tue, Oct 16, 2018 at 4:59 PM William Guo wrote: > > > > > Hi Bertrand, > > > > > > Thanks for the feedback! We will rephrase this in coming voting phase. > > > > > > William > > > > > > On Tue, Oct 16, 2018 at 4:25 PM Bertrand Delacretaz < > > > bdelacre...@codeconsult.ch> wrote: > > > > > > > Hi, > > > > > > > > On Tue, Oct 16, 2018 at 2:51 AM William Guo > wrote: > > > > > ...to establish a Project Management > > > > > Committee charged with the creation and maintenance of > > > > > open-source software, for distribution at no charge to > > > > > the public, related to a data quality solution for big data, > > > > > including both streaming and batch mode. It offers an unified > > > > > process to measure data quality from different perspectives... > > > > > > > > I suggest removing the last phrase, "It offers an unified...", I > think > > > > it's good to avoid being too specific in these descriptions. That > > > > phrase appears twice in the resolution, as usual. > > > > > > > > Apart from that, +1 on graduation! > > > > > > > > -Bertrand > > > > > > > > > >
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652822#comment-16652822
]
Eugene commented on GRIFFIN-200:
[~chemikadze]
thanks for your clarification here. I know the use case you mentioned.
you propose in the previous comment 'Updated potential interface of the hook to
have single universal method with different event subtypes', I agree with you.
so if we use event as type flag, do we need more listeners? maybe one general
interface could handle all listener outside.
about synchronous and asynchronous interface, I conclude my idea based on
common rule applying in distributed system, where asynchronous often is used
more than synchronous.
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface JobLifecycleHook {
> void onJobEvent(JobLifecycleEvent event) throws Exception;
> }
> public class BeforeJobCreated implements JobLifecycleEvent { ... }
> public class AfterJobCreated implements JobLifecycleEvent { ... }
> public class BeforeJobDeleted implements JobLifecycleEvent { ... }
> public class AfterJobDeleted implements JobLifecycleEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
Agree with William's proposal. Best Regards, Jenny On Tue, Oct 16, 2018 at 5:33 PM Lionel Liu wrote: > As I've commented in dev list, I agree with this proposal. > > Thanks, > Lionel > > On Tue, Oct 16, 2018 at 4:59 PM William Guo wrote: > > > Hi Bertrand, > > > > Thanks for the feedback! We will rephrase this in coming voting phase. > > > > William > > > > On Tue, Oct 16, 2018 at 4:25 PM Bertrand Delacretaz < > > bdelacre...@codeconsult.ch> wrote: > > > > > Hi, > > > > > > On Tue, Oct 16, 2018 at 2:51 AM William Guo wrote: > > > > ...to establish a Project Management > > > > Committee charged with the creation and maintenance of > > > > open-source software, for distribution at no charge to > > > > the public, related to a data quality solution for big data, > > > > including both streaming and batch mode. It offers an unified > > > > process to measure data quality from different perspectives... > > > > > > I suggest removing the last phrase, "It offers an unified...", I think > > > it's good to avoid being too specific in these descriptions. That > > > phrase appears twice in the resolution, as usual. > > > > > > Apart from that, +1 on graduation! > > > > > > -Bertrand > > > > > >
Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
As I've commented in dev list, I agree with this proposal. Thanks, Lionel On Tue, Oct 16, 2018 at 4:59 PM William Guo wrote: > Hi Bertrand, > > Thanks for the feedback! We will rephrase this in coming voting phase. > > William > > On Tue, Oct 16, 2018 at 4:25 PM Bertrand Delacretaz < > bdelacre...@codeconsult.ch> wrote: > > > Hi, > > > > On Tue, Oct 16, 2018 at 2:51 AM William Guo wrote: > > > ...to establish a Project Management > > > Committee charged with the creation and maintenance of > > > open-source software, for distribution at no charge to > > > the public, related to a data quality solution for big data, > > > including both streaming and batch mode. It offers an unified > > > process to measure data quality from different perspectives... > > > > I suggest removing the last phrase, "It offers an unified...", I think > > it's good to avoid being too specific in these descriptions. That > > phrase appears twice in the resolution, as usual. > > > > Apart from that, +1 on graduation! > > > > -Bertrand > > >
Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
Hi Bertrand, Thanks for the feedback! We will rephrase this in coming voting phase. William On Tue, Oct 16, 2018 at 4:25 PM Bertrand Delacretaz < bdelacre...@codeconsult.ch> wrote: > Hi, > > On Tue, Oct 16, 2018 at 2:51 AM William Guo wrote: > > ...to establish a Project Management > > Committee charged with the creation and maintenance of > > open-source software, for distribution at no charge to > > the public, related to a data quality solution for big data, > > including both streaming and batch mode. It offers an unified > > process to measure data quality from different perspectives... > > I suggest removing the last phrase, "It offers an unified...", I think > it's good to avoid being too specific in these descriptions. That > phrase appears twice in the resolution, as usual. > > Apart from that, +1 on graduation! > > -Bertrand >
Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
Hi, On Tue, Oct 16, 2018 at 2:51 AM William Guo wrote: > ...to establish a Project Management > Committee charged with the creation and maintenance of > open-source software, for distribution at no charge to > the public, related to a data quality solution for big data, > including both streaming and batch mode. It offers an unified > process to measure data quality from different perspectives... I suggest removing the last phrase, "It offers an unified...", I think it's good to avoid being too specific in these descriptions. That phrase appears twice in the resolution, as usual. Apart from that, +1 on graduation! -Bertrand
[jira] [Comment Edited] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651295#comment-16651295
]
Nikolay Sokolov edited comment on GRIFFIN-200 at 10/16/18 8:17 AM:
---
[~toyboxman] looks like there is slight confusion about what kind of events
this ticket is about. I'm talking here specifically about BatchJob/StreamingJob
entities and their lifecycle, not JobInstanceBean. For example, when new job is
created by POST /api/v1/jobs/, or disabled, or deleted. Idea here is to provide
mechanism to extend Griffin's behavior without forking it. For example:
* creating dashboards for each new job and/or alerts in third-party system
(Grafana, Elastalert)
* disabling alerts in third-party system if job is paused
* validating naming conventions on the jobs
* validating cron schedules (whether job is scheduled "not too frequently")
* RBAC implementation (Hive-style, as plugin)
Note that some of these use cases would need multiple listeners (to have
multiple listeners for multiple features at the same time), as well would
assume listeners to be synchronous (in a sense that handling is happening in
same thread, and can interrupt execution of operation). Good thing about
synchronous interface is that it allows listeners to have asynchronous behavior
internally, if required (have internal queueing, etc). On other hand
asynchronous semantics (let say, separate thread pool handling events) is not
possible to make behaving synchronously.
was (Author: chemikadze):
[~toyboxman] looks like there is slight confusion about what kind of events
this ticket is about. I'm talking here specifically about BatchJob/StreamingJob
entities and their lifecycle, not JobInstanceBean. For example, when new job is
created by POST /api/v1/jobs/, or disabled, or deleted. Idea here is to provide
mechanism to extend Griffin's behavior without forking it. For example:
* creating dashboards and/or alerts in third-party system (Grafana, Elastalert)
* disabling alerts in third-party system if job is paused
* validating naming conventions on the jobs
* validating cron schedules (whether job is scheduled "not too frequently")
* RBAC implementation (Hive-style, as plugin)
Note that some of these use cases would need multiple listeners (to have
multiple listeners for multiple features at the same time), as well would
assume listeners to be synchronous (in a sense that handling is happening in
same thread, and can interrupt execution of operation). Good thing about
synchronous interface is that it allows listeners to have asynchronous behavior
internally, if required (have internal queueing, etc). On other hand
asynchronous semantics (let say, separate thread pool handling events) is not
possible to make behaving synchronously.
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface JobLifecycleHook {
> void onJobEvent(JobLifecycleEvent event) throws Exception;
> }
> public class BeforeJobCreated implements JobLifecycleEvent { ... }
> public class AfterJobCreated implements JobLifecycleEvent { ... }
> public class BeforeJobDeleted implements JobLifecycleEvent { ... }
> public class AfterJobDeleted implements JobLifecycleEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651295#comment-16651295
]
Nikolay Sokolov commented on GRIFFIN-200:
-
[~toyboxman] looks like there is slight confusion about what kind of events
this ticket is about. I'm talking here specifically about BatchJob/StreamingJob
entities and their lifecycle, not JobInstanceBean. For example, when new job is
created by POST /api/v1/jobs/, or disabled, or deleted. Idea here is to provide
mechanism to extend Griffin's behavior without forking it. For example:
* creating dashboards and/or alerts in third-party system (Grafana, Elastalert)
* disabling alerts in third-party system if job is paused
* validating naming conventions on the jobs
* validating cron schedules (whether job is scheduled "not too frequently")
* RBAC implementation (Hive-style, as plugin)
Note that some of these use cases would need multiple listeners (to have
multiple listeners for multiple features at the same time), as well would
assume listeners to be synchronous (in a sense that handling is happening in
same thread, and can interrupt execution of operation). Good thing about
synchronous interface is that it allows listeners to have asynchronous behavior
internally, if required (have internal queueing, etc). On other hand
asynchronous semantics (let say, separate thread pool handling events) is not
possible to make behaving synchronously.
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface JobLifecycleHook {
> void onJobEvent(JobLifecycleEvent event) throws Exception;
> }
> public class BeforeJobCreated implements JobLifecycleEvent { ... }
> public class AfterJobCreated implements JobLifecycleEvent { ... }
> public class BeforeJobDeleted implements JobLifecycleEvent { ... }
> public class AfterJobDeleted implements JobLifecycleEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #439: Complement stuffs to docker guide.
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/439 ---
[GitHub] incubator-griffin issue #439: Complement stuffs to docker guide.
Github user guoyuepeng commented on the issue: https://github.com/apache/incubator-griffin/pull/439 LGTM ---
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651241#comment-16651241
]
Eugene commented on GRIFFIN-200:
[~guoyp] [~chemikadze]
I think it's better to take advantage of existed life cycle to implement
instead of new defined interface.
I don't know if livy job handle listener is enough to fulfill this requirement,
but it looks good.
[https://livy.incubator.apache.org/docs/latest/api/java/org/apache/livy/JobHandle.Listener.html]
about life cycle listener, do we need multiple listeners? what's use case for
multiple listeners, in my gut feeling one listener per one job is enough.
about synchronous or asynchronous, I prefer asynchronous on the grounds that in
a distributed system, we cannot assure synchronous call could return
considering networking/partition problems.
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface JobLifecycleHook {
> void onJobEvent(JobLifecycleEvent event) throws Exception;
> }
> public class BeforeJobCreated implements JobLifecycleEvent { ... }
> public class AfterJobCreated implements JobLifecycleEvent { ... }
> public class BeforeJobDeleted implements JobLifecycleEvent { ... }
> public class AfterJobDeleted implements JobLifecycleEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #438: update measure documents
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/438 ---
