[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063098#comment-16063098
]
Dawid Wysakowicz commented on FLINK-3414:
-
Basically what I want to propose here is to change the Pattern API completely,
that means both Scala and Java (which in my opinion should be based on the
Scala one). I think that answers both of your questions ;)
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062990#comment-16062990
]
Dian Fu commented on FLINK-3414:
Hi [~dawidwys], +1 to this proposal. Restricting illegal patterns at API level
is more user friendly. Two questions from my side:
# Why we need to port NFACompiler?
# Will we also consider the Java API?
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062863#comment-16062863
]
Dawid Wysakowicz commented on FLINK-3414:
-
Hi [~kkl0u], [~dian.fu]
As there is ongoing work to introduce GroupPatterns( [FLINK-6927] ) and
branching patterns ( [FLINK-4641] ) I would like to start a discussion about
reworking our Pattern API as a better hierarchical API would help in the
implementation.
I feel there is couple of problems with current API:
# there is lots of places where we throw {{MalformedPatternException}}, it
would be better to restrict illegal patterns in API rather than throw
exceptions in runtime
# API does not corresponds well to hierarchy of groups and branches
# it is hard to provide a good intuitive Scala dsl
In my branch [cep-api-rebuilt|
https://github.com/dawidwys/flink/tree/cep-api-rebuilt ] in {{flink-cep-scala}}
in package {{org.apache.flink.cep.scala.pattern.proto}} I created a proposal
for a new API written in Scala(better type support I think). For this API to
work we would need to port {{NFACompiler.java}} into the new API, which I think
should be done also in Scala, what would allow us to better leverage scala type
support by pattern matching.
As the current API is not annotated as Public, I feel there is no that much of
a problem with breaking API compatibility.
I would love to hear your opinions.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927063#comment-15927063
]
Ivan Mushketyk commented on FLINK-3414:
---
Hi [~dawidwys]
I don't have time to work on this task now, so I don't mind if you take over.
I'll reassign it to you.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897272#comment-15897272
]
Dawid Wysakowicz commented on FLINK-3414:
-
Hi [~ivan.mushketyk]
Do you still want to work on it or can I take over?
I really like your proposal for the API. I have some comments though. I would
not ommit the `where` keyword as it makes the code more readable and specify
the type in filter. So I would prefer:
{code}
pattern "start" where is[SubEvent] && _.id == 42 -> "next" where is[SubEvent]
&& _.getId() > 42
{code}
rather than
{code}
pattern[SubEvent] "start" _.id == 42 ->[SubEvent] _.getId() > 42
{code}
Also I would stick with the *within* as it betters describes the period.
I like better:
{code}
pattern "start" -> "middle" _.getName() == "error" ->> "end" _.getName() ==
"critical" within 10 seconds
{code}
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410706#comment-15410706
]
Ivan Mushketyk commented on FLINK-3414:
---
Hi [~till.rohrmann]
Do you have any thoughts about this proposal?
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404852#comment-15404852
]
Ivan Mushketyk commented on FLINK-3414:
---
I would like to describe my proposal on the case-by-case basis. Feel free to
point on any gaps and inconsistencies in this design.
Also, I am not sure what is the best medium for this kind of discussion. I will
start it here, but if you find it inconvenient or sub-optimal, I can move it to
some other place.
Let's start with a simplest pattern that expects a single event:
{code:java}
Pattern.begin("start").where(evt -> evt.getId() == 42)
{code}
This can be replaced with a much shorter Scala version
{code:java}
pattern "start" _.getId() == 42
{code}
A single function call *pattern* can replace *Pattern.begin* call and *where*
condition can be added as an argument to this function call.
A pattern that is expecting a specific type like this one:
{code:java}
Pattern.begin("start").where(evt -> evt.getId() == 42)
.subtype(SubEvent.class)
{code}
Can be replaced with the following Scala code:
{code:java}
pattern[SubEvent] "start" _.id == 42
{code}
The type of an object that is expected can be added as a type parameter to the
"pattern" function.
The *next* function can be replaced with the *->* operator:
{code:java}
Pattern.begin("start").where(evt -> evt.getId() == 42)
.subtype(SubEvent.class)
.next("next").where(evt -> evt.getId() > 42)
.subtype(SubEvent.class)
{code}
as in this code snippet:
{code:java}
pattern[SubEvent] "start" _.id == 42 ->[SubEvent] _.getId() > 42
{code}
or with a function call:
{code:java}
pattern[SubEvent] "start" _.id == 42 next[SubEvent] _.getId() > 42
{code}
A followedBy can be replaced with a *->>* operator:
{code:java}
Pattern.begin("start").where(evt -> evt.getId() == 42)
.next("middle").subtype(SubEvent.class).where(subEvt -> subEvt.getVolume()
>= 10.0)
.followedBy("end").where(evt -> evt.getName().equals("end"));
{code}
as in the following example:
{code:java}
pattern "start" _.getId() == 42
->[SubEvent] "middle" _.getVolume() >= 10.0
->> "end" _.getName() == "end"
{code}
A *within* function call can be replaced with *in* function that expects an
instance of a FinateDuration class that can provide a more readable code:
{code:java}
pattern "start" -> "middle" _.getName() == "error" ->> "end" _.getName() ==
"critical" in 10 seconds
{code}
h3. Additional operators
As far as I understand *or* and *not* operators are not yet implemented, but
here are few examples for these operators
{code:java}
// Pattern that has an event that is not SubEvent
pattern "start" ! is[SubEvent]
{code}
This example also shows an alternative way of specifying a type of element with
the *is* function.
{code:java}
// Pattern that has type SubEvent or has id 42
pattern "start" is[SubEvent] || is _.getId() == 42
{code}
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403931#comment-15403931
]
Ivan Mushketyk commented on FLINK-3414:
---
Thank you [~till.rohrmann], I think you are right.
I'll come up with the DSL draft in the next few days and write it here in the
comments.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403722#comment-15403722
]
Till Rohrmann commented on FLINK-3414:
--
Hi [~ivan.mushketyk],
I think it would be a good idea to first create a draft of the DSL before
investigating too much effort into the implementation. That way the community
can discuss whether the DSL makes sense and is useful without risking that you
have to make too big changes later on.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402825#comment-15402825
]
Ivan Mushketyk commented on FLINK-3414:
---
Hey Till.
Thank you for you reply.
I'll implement an initial version of the DSL and create a PR and then we can
continue discussing what should be changed in it.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Ivan Mushketyk
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402000#comment-15402000
]
Till Rohrmann commented on FLINK-3414:
--
Hi [~ivan.mushketyk], great to hear that you want to work on this issue.
The Scala support was partly added with FLINK-3708. However, we didn't add a
Scala DSL for the pattern definition yet. This means having a more succinct way
to specify in a more scalaesque way. The design for this language would be the
first step and should be discussed prior to starting the implementation.
The symbol "->" means "followedBy" and "||" is the or operator. The DSL example
does not have to represent the result of the design. It was intended to be an
inspiration how a future DSL could look like. So feel free to come up with your
own operator names to represent the different Pattern API concepts.
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3414) Add Scala API for CEP's pattern definition
[
https://issues.apache.org/jira/browse/FLINK-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401366#comment-15401366
]
Ivan Mushketyk commented on FLINK-3414:
---
Hi, I would like to work on this task if you don't mind.
Just a few clarification questions. What are "->" and "||" operators?
Are there any examples for the new API that you have in mind?
> Add Scala API for CEP's pattern definition
> --
>
> Key: FLINK-3414
> URL: https://issues.apache.org/jira/browse/FLINK-3414
> Project: Flink
> Issue Type: Improvement
> Components: CEP
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Priority: Minor
>
> Currently, the CEP library only supports a Java API to specify complex event
> patterns. In order to make it a bit less verbose for Scala users, it would be
> nice to also add a Scala API for the CEP library.
> A Scala API would also allow to pass Scala's anonymous functions as filter
> conditions or as a select function, for example, or to use partial functions
> to distinguish between different events.
> Furthermore, the Scala API could be designed to feel a bit more like a DSL:
> {code}
> begin "start" where _.id >= 42 -> "middle_1" as classOf[Subclass] ||
> "middle_2" where _.name equals "foobar" -> "end" where x => x.id <= x.volume
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
