Re: Flink SQL as DSL for flink CEP
Thanks a lot Jing . On Wednesday, June 16, 2021, 05:12:01 PM GMT+5:30, JING ZHANG wrote: Hi Dipanjan,I'm not an expert on Flink CEP, however I would like to share my immature ideas, hope it helps.Flink provides CEP library[1] which is a programmatic library which allows user define patterns based on pattern API.Besides, Flink allows user define patterns by MATCH_RECOGNIZE clause in SQL [2]. I think it's the right decision to use Flink SQL as a streaming CEP DSL. Please note most of the important features about CEP are already supported by Flink SQL, while a few features are not supported yet, you could view the document [2] to check. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/cep/[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sql/queries/match_recognize/ Best,JING ZHANG Dipanjan Mazumder 于2021年6月16日周三 下午12:44写道： Hi, Can we say that Flink SQL is kind of a DSL overlay on flink CEP , i mean i need a DSL for flink CEP , so that i can decouple the CEP rules from code and pass them dynamically to be applied on different data streams. Flink CEP doen't have any DSL implementation , so is it that Flink SQL can be used for the same purpose , where flink SQL has integration with Flink CEP and underlying flink SQL uses flink CEP for data processing. If that is the case can we use flink SQL as a streaming CEP DSL. RegardsDipanjan
Re: Flink SQL as DSL for flink CEP
Hi Dipanjan, I'm not an expert on Flink CEP, however I would like to share my immature ideas, hope it helps. Flink provides CEP library[1] which is a programmatic library which allows user define patterns based on pattern API. Besides, Flink allows user define patterns by MATCH_RECOGNIZE clause in SQL [2]. I think it's the right decision to use Flink SQL as a streaming CEP DSL. Please note most of the important features about CEP are already supported by Flink SQL, while a few features are not supported yet, you could view the document [2] to check. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/cep/ [2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sql/queries/match_recognize/ Best, JING ZHANG Dipanjan Mazumder 于2021年6月16日周三 下午12:44写道： > Hi, > > Can we say that Flink SQL is kind of a DSL overlay on flink CEP , i > mean i need a DSL for flink CEP , so that i can decouple the CEP rules from > code and pass them dynamically to be applied on different data streams. > Flink CEP doen't have any DSL implementation , so is it that Flink SQL can > be used for the same purpose , where flink SQL has integration with Flink > CEP and underlying flink SQL uses flink CEP for data processing. If that is > the case can we use flink SQL as a streaming CEP DSL. > > > Regards > Dipanjan >
Re: Flink SQL as DSL for flink CEP
Hi Dipanjan, Using Flink SQL's MATCH_RECOGNIZE operator is certainly a good idea if you are looking for a non-programmatic way to do CEP with Flink. On Wed, Jun 16, 2021 at 6:44 AM Dipanjan Mazumder wrote: > Hi, > > Can we say that Flink SQL is kind of a DSL overlay on flink CEP , i > mean i need a DSL for flink CEP , so that i can decouple the CEP rules from > code and pass them dynamically to be applied on different data streams. > Flink CEP doen't have any DSL implementation , so is it that Flink SQL can > be used for the same purpose , where flink SQL has integration with Flink > CEP and underlying flink SQL uses flink CEP for data processing. If that is > the case can we use flink SQL as a streaming CEP DSL. > > > Regards > Dipanjan >
Flink SQL as DSL for flink CEP
Hi, Can we say that Flink SQL is kind of a DSL overlay on flink CEP , i mean i need a DSL for flink CEP , so that i can decouple the CEP rules from code and pass them dynamically to be applied on different data streams. Flink CEP doen't have any DSL implementation , so is it that Flink SQL can be used for the same purpose , where flink SQL has integration with Flink CEP and underlying flink SQL uses flink CEP for data processing. If that is the case can we use flink SQL as a streaming CEP DSL. RegardsDipanjan
Re: DSL for Flink CEP
Thanks a lot ... On Thursday, June 3, 2021, 12:49:58 PM GMT+5:30, Dawid Wysakowicz wrote: Hi, Just to add on top to what Fabian said. The only community supported CEP library is the one that comes with Flink[1]. It is also used internally to support the MATCH_RECOGNIZE clause in Flink SQL[2]. Therefore there is a both programmatic library native DSL for defining patterns. Moreover you can use SQL. As Fabian mentioned, you can say the library is in a maintenance mode, primarily because there is no one interested in actively working on it. If you use other CEP libraries that are integrated with Flink, it's probably a better idea to reach out to the maintainers of said libraries. Lastly, I am not aware of any comparisons of CEP libraries/extensions that work with Flink. I am afraid you have to do the feature comparison yourself. I think the documentation for community supported library is a good start for it. Best, Dawid [1]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/cep/ [2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sql/queries/match_recognize/ On 02/06/2021 13:23, Fabian Paul wrote: Hi Dipanjan, I am afraid there are no foreseeable efforts planned but if you find a nice addition, you can start a discussion in the community about this feature. Best, Fabian On 2. Jun 2021, at 12:10, Dipanjan Mazumder wrote: Hi Fabian, Understood but is there any plan to grow the flink CEP and build a friendly DSL around flink CEP by any chance. Regards Dipanjan On Wednesday, June 2, 2021, 03:22:46 PM GMT+5:30, Fabian Paul wrote: Hi Dipanjan, Unfortunately, I have no experience with Siddhi but I am not aware of any official joined efforts between Flink and Siddhi. I can imagine that not all Siddhi CEP expressions are compatible with Flink’s CEP. At the moment there is also no active development for Flink’s CEP. I think to get a better understanding what the caveats are of the third party solution you have to directly reach out to the maintainers. Best, Fabian On 2. Jun 2021, at 08:37, Dipanjan Mazumder wrote: Hi , I am currently using Siddhi CEP with flink , but the flink-siddhi library has limited support for flnk versions and i will either need to fix the library or get tied to a fix version of Flink to use th library. I am looking at Flink CEP as an option , and also came across a Flink CEP DSL library (https://github.com/phil3k3/flink-cep-dsl) , but i am not sure about the acceptance for the same by the Flink community and DEV. Also is Flink CEP supporting the Siddhi CEP constructs and is rich on the same aspect. Please let me know the same , so that i can take a cautious decision on the same. Regards Dipanjan
Re: DSL for Flink CEP
Hi, Just to add on top to what Fabian said. The only community supported CEP library is the one that comes with Flink[1]. It is also used internally to support the MATCH_RECOGNIZE clause in Flink SQL[2]. Therefore there is a both programmatic library native DSL for defining patterns. Moreover you can use SQL. As Fabian mentioned, you can say the library is in a maintenance mode, primarily because there is no one interested in actively working on it. If you use other CEP libraries that are integrated with Flink, it's probably a better idea to reach out to the maintainers of said libraries. Lastly, I am not aware of any comparisons of CEP libraries/extensions that work with Flink. I am afraid you have to do the feature comparison yourself. I think the documentation for community supported library is a good start for it. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/cep/ [2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sql/queries/match_recognize/ On 02/06/2021 13:23, Fabian Paul wrote: > Hi Dipanjan, > > I am afraid there are no foreseeable efforts planned but if you find a > nice addition, you can > start a discussion in the community about this feature. > > Best, > Fabian >> On 2. Jun 2021, at 12:10, Dipanjan Mazumder > <mailto:java...@yahoo.com>> wrote: >> >> Hi Fabian, >> >> Understood but is there any plan to grow the flink CEP and >> build a friendly DSL around flink CEP by any chance. >> >> Regards >> Dipanjan >> >> On Wednesday, June 2, 2021, 03:22:46 PM GMT+5:30, Fabian Paul >> mailto:fabianp...@data-artisans.com>> >> wrote: >> >> >> Hi Dipanjan, >> >> Unfortunately, I have no experience with Siddhi but I am not aware of >> any official joined efforts between Flink and Siddhi. >> I can imagine that not all Siddhi CEP expressions are compatible with >> Flink’s CEP. At the moment there is also no active >> development for Flink’s CEP. >> >> I think to get a better understanding what the caveats are of the >> third party solution you have to directly reach out to the >> maintainers. >> >> Best, >> Fabian >> >> >>> On 2. Jun 2021, at 08:37, Dipanjan Mazumder >> <mailto:java...@yahoo.com>> wrote: >>> >>> Hi , >>> >>> I am currently using Siddhi CEP with flink , but the flink-siddhi >>> library has limited support for flnk versions and i will either need >>> to fix the library or get tied to a fix version of Flink to use th >>> library. >>> >>> I am looking at Flink CEP as an option , and also came across a >>> Flink CEP DSL library (https://github.com/phil3k3/flink-cep-dsl >>> <https://github.com/phil3k3/flink-cep-dsl>) , but i am not sure >>> about the acceptance for the same by the Flink community and DEV. >>> Also is Flink CEP supporting the Siddhi CEP constructs and is rich >>> on the same aspect. >>> >>> Please let me know the same , so that i can take a cautious decision >>> on the same. >>> >>> Regards >>> Dipanjan >> > OpenPGP_signature Description: OpenPGP digital signature
Re: DSL for Flink CEP
Hi Dipanjan, I am afraid there are no foreseeable efforts planned but if you find a nice addition, you can start a discussion in the community about this feature. Best, Fabian > On 2. Jun 2021, at 12:10, Dipanjan Mazumder wrote: > > Hi Fabian, > > Understood but is there any plan to grow the flink CEP and build a > friendly DSL around flink CEP by any chance. > > Regards > Dipanjan > > On Wednesday, June 2, 2021, 03:22:46 PM GMT+5:30, Fabian Paul > wrote: > > > Hi Dipanjan, > > Unfortunately, I have no experience with Siddhi but I am not aware of any > official joined efforts between Flink and Siddhi. > I can imagine that not all Siddhi CEP expressions are compatible with Flink’s > CEP. At the moment there is also no active > development for Flink’s CEP. > > I think to get a better understanding what the caveats are of the third party > solution you have to directly reach out to the > maintainers. > > Best, > Fabian > > >> On 2. Jun 2021, at 08:37, Dipanjan Mazumder > <mailto:java...@yahoo.com>> wrote: >> >> Hi , >> >>I am currently using Siddhi CEP with flink , but the flink-siddhi library >> has limited support for flnk versions and i will either need to fix the >> library or get tied to a fix version of Flink to use th library. >> >> I am looking at Flink CEP as an option , and also came across a Flink CEP >> DSL library (https://github.com/phil3k3/flink-cep-dsl >> <https://github.com/phil3k3/flink-cep-dsl>) , but i am not sure about the >> acceptance for the same by the Flink community and DEV. Also is Flink CEP >> supporting the Siddhi CEP constructs and is rich on the same aspect. >> >> Please let me know the same , so that i can take a cautious decision on the >> same. >> >> Regards >> Dipanjan >
Re: DSL for Flink CEP
Hi Fabian, Understood but is there any plan to grow the flink CEP and build a friendly DSL around flink CEP by any chance. RegardsDipanjan On Wednesday, June 2, 2021, 03:22:46 PM GMT+5:30, Fabian Paul wrote: Hi Dipanjan, Unfortunately, I have no experience with Siddhi but I am not aware of any official joined efforts between Flink and Siddhi.I can imagine that not all Siddhi CEP expressions are compatible with Flink’s CEP. At the moment there is also no activedevelopment for Flink’s CEP. I think to get a better understanding what the caveats are of the third party solution you have to directly reach out to the maintainers. Best,Fabian On 2. Jun 2021, at 08:37, Dipanjan Mazumder wrote: Hi , I am currently using Siddhi CEP with flink , but the flink-siddhi library has limited support for flnk versions and i will either need to fix the library or get tied to a fix version of Flink to use th library. I am looking at Flink CEP as an option , and also came across a Flink CEP DSL library (https://github.com/phil3k3/flink-cep-dsl) , but i am not sure about the acceptance for the same by the Flink community and DEV. Also is Flink CEP supporting the Siddhi CEP constructs and is rich on the same aspect. Please let me know the same , so that i can take a cautious decision on the same. RegardsDipanjan
Re: DSL for Flink CEP
Hi Dipanjan, Unfortunately, I have no experience with Siddhi but I am not aware of any official joined efforts between Flink and Siddhi. I can imagine that not all Siddhi CEP expressions are compatible with Flink’s CEP. At the moment there is also no active development for Flink’s CEP. I think to get a better understanding what the caveats are of the third party solution you have to directly reach out to the maintainers. Best, Fabian > On 2. Jun 2021, at 08:37, Dipanjan Mazumder wrote: > > Hi , > >I am currently using Siddhi CEP with flink , but the flink-siddhi library > has limited support for flnk versions and i will either need to fix the > library or get tied to a fix version of Flink to use th library. > > I am looking at Flink CEP as an option , and also came across a Flink CEP > DSL library (https://github.com/phil3k3/flink-cep-dsl) , but i am not sure > about the acceptance for the same by the Flink community and DEV. Also is > Flink CEP supporting the Siddhi CEP constructs and is rich on the same aspect. > > Please let me know the same , so that i can take a cautious decision on the > same. > > Regards > Dipanjan
DSL for Flink CEP
Hi , I am currently using Siddhi CEP with flink , but the flink-siddhi library has limited support for flnk versions and i will either need to fix the library or get tied to a fix version of Flink to use th library. I am looking at Flink CEP as an option , and also came across a Flink CEP DSL library (https://github.com/phil3k3/flink-cep-dsl) , but i am not sure about the acceptance for the same by the Flink community and DEV. Also is Flink CEP supporting the Siddhi CEP constructs and is rich on the same aspect. Please let me know the same , so that i can take a cautious decision on the same. RegardsDipanjan
