[GitHub] incubator-griffin pull request #442: Support "cache" field on rules in servi...
Github user chemikadze commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/442#discussion_r227217198 --- Diff: service/src/main/java/org/apache/griffin/core/measure/entity/Rule.java --- @@ -81,6 +81,9 @@ Licensed to the Apache Software Foundation (ASF) under one @Column(name = "\"out\"") private String out; +@JsonInclude(JsonInclude.Include.NON_NULL) --- End diff -- @toyboxman don't see anything, service/api-guide.md does not describe any style guides or standard contracts, only shows examples. Also "cache" field was not documented in measure-configuration-guide.md, added information about it there. Javadoc says: ``` * @param cache cache the result for multiple usage (optional, valid for "spark-sql" and "df-ops" mode) ``` Taking into account it's valid not for all types of rules, hiding null values seems to be only valid behavior. ---
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660037#comment-16660037
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227214831
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
@whhe fixed, thanks
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #441: [GRIFFIN-207] LDAP login service improv...
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227214831
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
@whhe fixed, thanks
---
Re: Hooks support in JobService
LGTM, we can start this. Thanks, Lionel On Mon, Oct 22, 2018 at 8:58 PM Eugene Liu wrote: > I'm fine this hook proposal. > > one comment, could we define annotation to take list of hook class names > as well. > > > From: William Guo > Sent: Monday, October 22, 2018 3:42 PM > To: dev@griffin.incubator.apache.org > Subject: Re: Hooks support in JobService > > Sounds good to me. > >- application.properties would have a property containing list of hook >class names (or probably, spring bean names?), configuring subset and > order >of hook instances enabled at deployment time. > [William] I like a logic name as spring bean name. > >- Hook instantiation mechanism should create them as proper spring >beans, so that they could use same property file for configuration, and >access existing repositories and services. At the same time, only >explicitly enabled hooks should be instantiated, to avoid initializing > any >optional heavy hooks each time. > [William] We can launch our hook mechanism by leveraging spring bean > instantializtion mechanism. > >- Hooks would be implementing GriffinHook interface, and particular >implementation would use any kind of internal logic, from spring >integration to some asynchronous handling. > [William] Sounds good to me. > >- Hook interface would have single method accepting subclass of >GriffinHookEvent. Hook implementation would pick events it would like to >react on, and ignore the others. To give it some structure, events would > be >organized in hierarchy, like JobEvents, JobInstanceEvents, > MeasureEvents, >etc. > [William] Events are common parts for the Event driven flow. > >- New extension points would be added by introducing new types of events >corresponding to some point inside griffin services. Service instance > would >be creating corresponding event object, and call onEvent method on all > hook >instances defined in .properties file, according to defined order. > [William] Let's start with basic events, get it run in prototype, we can > refine this later. > > > > On Mon, Oct 22, 2018 at 2:51 PM Nick Sokolov wrote: > > > I think this is best detail I can give right now without writing any > code: > > > >- application.properties would have a property containing list of hook > >class names (or probably, spring bean names?), configuring subset and > > order > >of hook instances enabled at deployment time. > >- Hook instantiation mechanism should create them as proper spring > >beans, so that they could use same property file for configuration, > and > >access existing repositories and services. At the same time, only > >explicitly enabled hooks should be instantiated, to avoid initializing > > any > >optional heavy hooks each time. > >- Hooks would be implementing GriffinHook interface, and particular > >implementation would use any kind of internal logic, from spring > >integration to some asynchronous handling. > >- Hook interface would have single method accepting subclass of > >GriffinHookEvent. Hook implementation would pick events it would like > to > >react on, and ignore the others. To give it some structure, events > > would be > >organized in hierarchy, like JobEvents, JobInstanceEvents, > > MeasureEvents, > >etc. > >- New extension points would be added by introducing new types of > events > >corresponding to some point inside griffin services. Service instance > > would > >be creating corresponding event object, and call onEvent method on all > > hook > >instances defined in .properties file, according to defined order. > > > > Probably it's time to build some technical prototype. > > > > On Sun, Oct 21, 2018 at 6:46 PM Eugene Liu wrote: > > > > > Guys, > > > > > > you have agree to integration style that keeps two potential solutions, > > > Hive/Spring, leaves final decision to users. I'm fine on this > agreement. > > > > > > but do you have comprehensive design schema, I think the topic > discussion > > > is not closing in jira, we should keep in same page about > > > event/message/sync/async details. > > > > > > thanks > > > > > > From: William Guo > > > Sent: Monday, October 22, 2018 8:26 AM > > > To: dev@griffin.incubator.apache.org > > > Subject: Re: Hooks support in JobService > > > > > > Sounds good to me. > > > > > > You can implement your preferred hive-style, I can implement spring > > > integration based on your interface. > > > > > > William > > > > > > On Mon, Oct 22, 2018 at 2:10 AM Nick Sokolov > > wrote: > > > > > > > Hi William, > > > > > > > > Totally agree on keeping interface as abstract as possible. > Interfaces > > > feel > > > > like natural solution here. > > > > It should be relatively easy to provide spring integration > > implementation > > > > of hook interface. > > > > Probably we can even have
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659926#comment-16659926
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user whhe commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227194519
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
It is not recommended to throw RuntimeException directly in Griffin, maybe
you can use ServiceException from GriffinException class instead.
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #441: [GRIFFIN-207] LDAP login service improv...
Github user whhe commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227194519
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
It is not recommended to throw RuntimeException directly in Griffin, maybe
you can use ServiceException from GriffinException class instead.
---
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659533#comment-16659533
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/441
@toyboxman i've updated the code according to your comments, thanks!
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin issue #441: [GRIFFIN-207] LDAP login service improvements
Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/441 @toyboxman i've updated the code according to your comments, thanks! ---
[jira] [Created] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
Nikolay Sokolov created GRIFFIN-208:
---
Summary: Job status is SUCCESS even if some stages have failed
Key: GRIFFIN-208
URL: https://issues.apache.org/jira/browse/GRIFFIN-208
Project: Griffin (Incubating)
Issue Type: Bug
Reporter: Nikolay Sokolov
When some steps (MetricWrite or SparkSql, for example) fail, errors are just
logged, but not reported as part of job status. Symptoms:
{code:none}
18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [ ]
error: ...
{code}
YarnApplicationState: FINISHED
FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Hooks support in JobService
I'm fine this hook proposal. one comment, could we define annotation to take list of hook class names as well. From: William Guo Sent: Monday, October 22, 2018 3:42 PM To: dev@griffin.incubator.apache.org Subject: Re: Hooks support in JobService Sounds good to me. - application.properties would have a property containing list of hook class names (or probably, spring bean names?), configuring subset and order of hook instances enabled at deployment time. [William] I like a logic name as spring bean name. - Hook instantiation mechanism should create them as proper spring beans, so that they could use same property file for configuration, and access existing repositories and services. At the same time, only explicitly enabled hooks should be instantiated, to avoid initializing any optional heavy hooks each time. [William] We can launch our hook mechanism by leveraging spring bean instantializtion mechanism. - Hooks would be implementing GriffinHook interface, and particular implementation would use any kind of internal logic, from spring integration to some asynchronous handling. [William] Sounds good to me. - Hook interface would have single method accepting subclass of GriffinHookEvent. Hook implementation would pick events it would like to react on, and ignore the others. To give it some structure, events would be organized in hierarchy, like JobEvents, JobInstanceEvents, MeasureEvents, etc. [William] Events are common parts for the Event driven flow. - New extension points would be added by introducing new types of events corresponding to some point inside griffin services. Service instance would be creating corresponding event object, and call onEvent method on all hook instances defined in .properties file, according to defined order. [William] Let's start with basic events, get it run in prototype, we can refine this later. On Mon, Oct 22, 2018 at 2:51 PM Nick Sokolov wrote: > I think this is best detail I can give right now without writing any code: > >- application.properties would have a property containing list of hook >class names (or probably, spring bean names?), configuring subset and > order >of hook instances enabled at deployment time. >- Hook instantiation mechanism should create them as proper spring >beans, so that they could use same property file for configuration, and >access existing repositories and services. At the same time, only >explicitly enabled hooks should be instantiated, to avoid initializing > any >optional heavy hooks each time. >- Hooks would be implementing GriffinHook interface, and particular >implementation would use any kind of internal logic, from spring >integration to some asynchronous handling. >- Hook interface would have single method accepting subclass of >GriffinHookEvent. Hook implementation would pick events it would like to >react on, and ignore the others. To give it some structure, events > would be >organized in hierarchy, like JobEvents, JobInstanceEvents, > MeasureEvents, >etc. >- New extension points would be added by introducing new types of events >corresponding to some point inside griffin services. Service instance > would >be creating corresponding event object, and call onEvent method on all > hook >instances defined in .properties file, according to defined order. > > Probably it's time to build some technical prototype. > > On Sun, Oct 21, 2018 at 6:46 PM Eugene Liu wrote: > > > Guys, > > > > you have agree to integration style that keeps two potential solutions, > > Hive/Spring, leaves final decision to users. I'm fine on this agreement. > > > > but do you have comprehensive design schema, I think the topic discussion > > is not closing in jira, we should keep in same page about > > event/message/sync/async details. > > > > thanks > > > > From: William Guo > > Sent: Monday, October 22, 2018 8:26 AM > > To: dev@griffin.incubator.apache.org > > Subject: Re: Hooks support in JobService > > > > Sounds good to me. > > > > You can implement your preferred hive-style, I can implement spring > > integration based on your interface. > > > > William > > > > On Mon, Oct 22, 2018 at 2:10 AM Nick Sokolov > wrote: > > > > > Hi William, > > > > > > Totally agree on keeping interface as abstract as possible. Interfaces > > feel > > > like natural solution here. > > > It should be relatively easy to provide spring integration > implementation > > > of hook interface. > > > Probably we can even have default spring integration hook provided as > an > > > example. > > > > > > On Sun, Oct 21, 2018 at 6:33 AM William Guo wrote: > > > > > > > hi Nick, > > > > > > > > I agree with you we need to have some lifecycle hooks for jobs. > > > > > > > > No matter which extension mechanism we choose at runtime, > > > > but we can always abstract
[GitHub] incubator-griffin pull request #440: Update recommended ES index schema
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/440 ---
Re: Hooks support in JobService
Sounds good to me. - application.properties would have a property containing list of hook class names (or probably, spring bean names?), configuring subset and order of hook instances enabled at deployment time. [William] I like a logic name as spring bean name. - Hook instantiation mechanism should create them as proper spring beans, so that they could use same property file for configuration, and access existing repositories and services. At the same time, only explicitly enabled hooks should be instantiated, to avoid initializing any optional heavy hooks each time. [William] We can launch our hook mechanism by leveraging spring bean instantializtion mechanism. - Hooks would be implementing GriffinHook interface, and particular implementation would use any kind of internal logic, from spring integration to some asynchronous handling. [William] Sounds good to me. - Hook interface would have single method accepting subclass of GriffinHookEvent. Hook implementation would pick events it would like to react on, and ignore the others. To give it some structure, events would be organized in hierarchy, like JobEvents, JobInstanceEvents, MeasureEvents, etc. [William] Events are common parts for the Event driven flow. - New extension points would be added by introducing new types of events corresponding to some point inside griffin services. Service instance would be creating corresponding event object, and call onEvent method on all hook instances defined in .properties file, according to defined order. [William] Let's start with basic events, get it run in prototype, we can refine this later. On Mon, Oct 22, 2018 at 2:51 PM Nick Sokolov wrote: > I think this is best detail I can give right now without writing any code: > >- application.properties would have a property containing list of hook >class names (or probably, spring bean names?), configuring subset and > order >of hook instances enabled at deployment time. >- Hook instantiation mechanism should create them as proper spring >beans, so that they could use same property file for configuration, and >access existing repositories and services. At the same time, only >explicitly enabled hooks should be instantiated, to avoid initializing > any >optional heavy hooks each time. >- Hooks would be implementing GriffinHook interface, and particular >implementation would use any kind of internal logic, from spring >integration to some asynchronous handling. >- Hook interface would have single method accepting subclass of >GriffinHookEvent. Hook implementation would pick events it would like to >react on, and ignore the others. To give it some structure, events > would be >organized in hierarchy, like JobEvents, JobInstanceEvents, > MeasureEvents, >etc. >- New extension points would be added by introducing new types of events >corresponding to some point inside griffin services. Service instance > would >be creating corresponding event object, and call onEvent method on all > hook >instances defined in .properties file, according to defined order. > > Probably it's time to build some technical prototype. > > On Sun, Oct 21, 2018 at 6:46 PM Eugene Liu wrote: > > > Guys, > > > > you have agree to integration style that keeps two potential solutions, > > Hive/Spring, leaves final decision to users. I'm fine on this agreement. > > > > but do you have comprehensive design schema, I think the topic discussion > > is not closing in jira, we should keep in same page about > > event/message/sync/async details. > > > > thanks > > > > From: William Guo > > Sent: Monday, October 22, 2018 8:26 AM > > To: dev@griffin.incubator.apache.org > > Subject: Re: Hooks support in JobService > > > > Sounds good to me. > > > > You can implement your preferred hive-style, I can implement spring > > integration based on your interface. > > > > William > > > > On Mon, Oct 22, 2018 at 2:10 AM Nick Sokolov > wrote: > > > > > Hi William, > > > > > > Totally agree on keeping interface as abstract as possible. Interfaces > > feel > > > like natural solution here. > > > It should be relatively easy to provide spring integration > implementation > > > of hook interface. > > > Probably we can even have default spring integration hook provided as > an > > > example. > > > > > > On Sun, Oct 21, 2018 at 6:33 AM William Guo wrote: > > > > > > > hi Nick, > > > > > > > > I agree with you we need to have some lifecycle hooks for jobs. > > > > > > > > No matter which extension mechanism we choose at runtime, > > > > but we can always abstract common lifecycle interfaces, and > implements > > > > different solutions for different environments. > > > > > > > > Say, at runtime someone can configure griffin extension mechanism as > > > > 'Hive-style' and others can configure as spring integration model. > > > > > > > > We cannot
Re: Hooks support in JobService
I think this is best detail I can give right now without writing any code:
- application.properties would have a property containing list of hook
class names (or probably, spring bean names?), configuring subset and order
of hook instances enabled at deployment time.
- Hook instantiation mechanism should create them as proper spring
beans, so that they could use same property file for configuration, and
access existing repositories and services. At the same time, only
explicitly enabled hooks should be instantiated, to avoid initializing any
optional heavy hooks each time.
- Hooks would be implementing GriffinHook interface, and particular
implementation would use any kind of internal logic, from spring
integration to some asynchronous handling.
- Hook interface would have single method accepting subclass of
GriffinHookEvent. Hook implementation would pick events it would like to
react on, and ignore the others. To give it some structure, events would be
organized in hierarchy, like JobEvents, JobInstanceEvents, MeasureEvents,
etc.
- New extension points would be added by introducing new types of events
corresponding to some point inside griffin services. Service instance would
be creating corresponding event object, and call onEvent method on all hook
instances defined in .properties file, according to defined order.
Probably it's time to build some technical prototype.
On Sun, Oct 21, 2018 at 6:46 PM Eugene Liu wrote:
> Guys,
>
> you have agree to integration style that keeps two potential solutions,
> Hive/Spring, leaves final decision to users. I'm fine on this agreement.
>
> but do you have comprehensive design schema, I think the topic discussion
> is not closing in jira, we should keep in same page about
> event/message/sync/async details.
>
> thanks
>
> From: William Guo
> Sent: Monday, October 22, 2018 8:26 AM
> To: dev@griffin.incubator.apache.org
> Subject: Re: Hooks support in JobService
>
> Sounds good to me.
>
> You can implement your preferred hive-style, I can implement spring
> integration based on your interface.
>
> William
>
> On Mon, Oct 22, 2018 at 2:10 AM Nick Sokolov wrote:
>
> > Hi William,
> >
> > Totally agree on keeping interface as abstract as possible. Interfaces
> feel
> > like natural solution here.
> > It should be relatively easy to provide spring integration implementation
> > of hook interface.
> > Probably we can even have default spring integration hook provided as an
> > example.
> >
> > On Sun, Oct 21, 2018 at 6:33 AM William Guo wrote:
> >
> > > hi Nick,
> > >
> > > I agree with you we need to have some lifecycle hooks for jobs.
> > >
> > > No matter which extension mechanism we choose at runtime,
> > > but we can always abstract common lifecycle interfaces, and implements
> > > different solutions for different environments.
> > >
> > > Say, at runtime someone can configure griffin extension mechanism as
> > > 'Hive-style' and others can configure as spring integration model.
> > >
> > > We cannot decide which one is super than other, but we can implement
> > > multiply solutions and let users decide at deployment time?
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > William
> > >
> > >
> > >
> > > On Fri, Oct 19, 2018 at 2:03 PM Nick Sokolov
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like invite the community to discussion of Job object lifecycle
> > hooks
> > > > design (GRIFFIN-200 <
> https://issues.apache.org/jira/browse/GRIFFIN-200
> > > >).
> > > >
> > > > On a high level, idea is to have hooks executed triggered by
> JobService
> > > > before/after job is created, or before/after job is deleted, and so
> on.
> > > > Main point is to provide extension mechanism for JobService, allowing
> > > > third-party implementations of following features:
> > > >
> > > >- validating naming conventions on the jobs
> > > >- validating cron schedules (whether job is scheduled "not too
> > > >frequently")
> > > >- creating dashboards for each new job and/or alerts in
> third-party
> > > >system (Grafana, Elastalert)
> > > >- RBAC implementation (Hive-style, as plugin)
> > > >- disabling alerts in third-party system if job is paused
> > > >
> > > > Right now several aspects are to be decided:
> > > >
> > > >1. particular extention mehanism (POJO interface implementations
> > > >("Hive-style"), or Spring-Integration, etc)
> > > >2. whether only one or multiple hooks could be configured at same
> > time
> > > >3. execution model: whether they should run synchronously, or
> > > >asynchronously; sequentially, or in parallel
> > > >
> > > > Ticket already has some discussion going, so inviting everyone to
> Jira
> > > > ticket, to avoid having same conversation in two places.
> > > >
> > >
> >
>
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658634#comment-16658634
]
Nikolay Sokolov commented on GRIFFIN-200:
-
[~Lionel_3L] thank you, expanding scope of hook events beyond just Job
instances is very good point. Updated signature in a ticket.
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface GriffinHook {
> void onEvent(GriffinHookEvent event) throws Exception;
> }
> public interface GriffinHookEvent { ... }
> public interface JobEvent implements GriffinHookEvent { ... }
> public class BeforeJobCreated implements JobEvent { ... }
> public class AfterJobCreated implements JobEvent { ... }
> public class BeforeJobDeleted implements JobEvent { ... }
> public class AfterJobDeleted implements JobEvent { ... }
> public interface JobInstanceEvent implements GriffinHookEvent { ... }
> public class BeforeJobInstanceStart implements JobInstanceEvent { ... }
> public class AfterJobInstanceEnd implements JobInstanceEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Sokolov updated GRIFFIN-200:
Description:
In some environments, users might want to perform certain actions before/after
job is created, before/after job is activated, before/after job is deleted, and
so on.
To fullfill that need, some hook plugin mechanism can be provided, similar to
what Hive is doing. User would place respective jar files into Service module
classpath at deployment time, and would specify class names using some
annotation or using property listing class names (particular mechanism is yet
to be determined).
Proposed signature:
{code:none}
public interface GriffinHook {
void onEvent(GriffinHookEvent event) throws Exception;
}
public interface GriffinHookEvent { ... }
public interface JobEvent implements GriffinHookEvent { ... }
public class BeforeJobCreated implements JobEvent { ... }
public class AfterJobCreated implements JobEvent { ... }
public class BeforeJobDeleted implements JobEvent { ... }
public class AfterJobDeleted implements JobEvent { ... }
public interface JobInstanceEvent implements GriffinHookEvent { ... }
public class BeforeJobInstanceStart implements JobInstanceEvent { ... }
public class AfterJobInstanceEnd implements JobInstanceEvent { ... }
{code}
was:
In some environments, users might want to perform certain actions before/after
job is created, before/after job is activated, before/after job is deleted, and
so on.
To fullfill that need, some hook plugin mechanism can be provided, similar to
what Hive is doing. User would place respective jar files into Service module
classpath at deployment time, and would specify class names using some
annotation or using property listing class names (particular mechanism is yet
to be determined).
Proposed signature:
{code:none}
public interface JobLifecycleHook {
void onJobEvent(JobLifecycleEvent event) throws Exception;
}
public class BeforeJobCreated implements JobLifecycleEvent { ... }
public class AfterJobCreated implements JobLifecycleEvent { ... }
public class BeforeJobDeleted implements JobLifecycleEvent { ... }
public class AfterJobDeleted implements JobLifecycleEvent { ... }
{code}
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface GriffinHook {
> void onEvent(GriffinHookEvent event) throws Exception;
> }
> public interface GriffinHookEvent { ... }
> public interface JobEvent implements GriffinHookEvent { ... }
> public class BeforeJobCreated implements JobEvent { ... }
> public class AfterJobCreated implements JobEvent { ... }
> public class BeforeJobDeleted implements JobEvent { ... }
> public class AfterJobDeleted implements JobEvent { ... }
> public interface JobInstanceEvent implements GriffinHookEvent { ... }
> public class BeforeJobInstanceStart implements JobInstanceEvent { ... }
> public class AfterJobInstanceEnd implements JobInstanceEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
