Re: Spark streaming for CEP
Thanks very much Mich, Thomas and Stephan . I will look into it. On Tue, Oct 24, 2017 at 8:02 PM, lucas.g...@gmail.com <lucas.g...@gmail.com> wrote: > This looks really interesting, thanks for linking! > > Gary Lucas > > On 24 October 2017 at 15:06, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Great thanks Steve >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 24 October 2017 at 22:58, Stephen Boesch <java...@gmail.com> wrote: >> >>> Hi Mich, the github link has a brief intro - including a link to the >>> formal docs http://logisland.readthedocs.io/en/latest/index.html . >>> They have an architectural overview, developer guide, tutorial, and pretty >>> comprehensive api docs. >>> >>> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >>> >>>> thanks Thomas. >>>> >>>> do you have a summary write-up for this tool please? >>>> >>>> >>>> regards, >>>> >>>> >>>> >>>> >>>> Thomas >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 24 October 2017 at 13:53, Thomas Bailet <thomas.bai...@hurence.com> >>>> wrote: >>>> >>>>> Hi >>>>> >>>>> we (@ hurence) have released on open source middleware based on >>>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland* >>>>> (https://github.com/Hurence/logisland/) it has been deployed into >>>>> production for 2 years now and does a great job. You should have a look. >>>>> >>>>> >>>>> bye >>>>> >>>>> Thomas Bailet >>>>> >>>>> CTO : hurence >>>>> >>>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : >>>>> >>>>> As you may be aware the granularity that Spark streaming has is >>>>> micro-batching and that is limited to 0.5 second. So if you have >>>>> continuous >>>>> ingestion of data then Spark streaming may not be granular enough for CEP. >>>>> You may consider other products. >>>>> >>>>> Worth looking at this old thread on mine "Spark support for Complex >>>>> Event Processing (CEP) >>>>> >>>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604. >>>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@ >>>>> mail.gmail.com%3E >>>>> >>>>> HTH >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: >>>>> >>>>>> Hello all, >>>>>> >>>>>> Has anyone used spark streaming for CEP (Complex Event processing). >>>>>> Any CEP libraries that works well with spark. I have a use case for CEP >>>>>> and >>>>>> trying to see if spark streaming is a good fit. >>>>>> >>>>>> Currently we have a data pipeline using Kafka, Spark streaming and >>>>>> Cassandra for data ingestion and near real time dashboard. >>>>>> >>>>>> Please share your experience. >>>>>> Thanks much. >>>>>> -Anna >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
Re: Spark streaming for CEP
This looks really interesting, thanks for linking! Gary Lucas On 24 October 2017 at 15:06, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Great thanks Steve > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 24 October 2017 at 22:58, Stephen Boesch <java...@gmail.com> wrote: > >> Hi Mich, the github link has a brief intro - including a link to the >> formal docs http://logisland.readthedocs.io/en/latest/index.html . >> They have an architectural overview, developer guide, tutorial, and pretty >> comprehensive api docs. >> >> 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >> >>> thanks Thomas. >>> >>> do you have a summary write-up for this tool please? >>> >>> >>> regards, >>> >>> >>> >>> >>> Thomas >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 24 October 2017 at 13:53, Thomas Bailet <thomas.bai...@hurence.com> >>> wrote: >>> >>>> Hi >>>> >>>> we (@ hurence) have released on open source middleware based on >>>> SparkStreaming over Kafka to do CEP and log mining, called *logisland* >>>> (https://github.com/Hurence/logisland/) it has been deployed into >>>> production for 2 years now and does a great job. You should have a look. >>>> >>>> >>>> bye >>>> >>>> Thomas Bailet >>>> >>>> CTO : hurence >>>> >>>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : >>>> >>>> As you may be aware the granularity that Spark streaming has is >>>> micro-batching and that is limited to 0.5 second. So if you have continuous >>>> ingestion of data then Spark streaming may not be granular enough for CEP. >>>> You may consider other products. >>>> >>>> Worth looking at this old thread on mine "Spark support for Complex >>>> Event Processing (CEP) >>>> >>>> https://mail-archives.apache.org/mod_mbox/spark-user/201604. >>>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@ >>>> mail.gmail.com%3E >>>> >>>> HTH >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: >>>> >>>>> Hello all, >>>>> >>>>> Has anyone used spark streaming for CEP (Complex Event processing). >>>>> Any CEP libraries that works well with spark. I have a use case for CEP >>>>> and >>>>> trying to see if spark streaming is a good fit. >>>>> >>>>> Currently we have a data pipeline using Kafka, Spark streaming and >>>>> Cassandra for data ingestion and near real time dashboard. >>>>> >>>>> Please share your experience. >>>>> Thanks much. >>>>> -Anna >>>>> >>>>> >>>>> >>>> >>>> >>> >> >
Re: Spark streaming for CEP
Great thanks Steve Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 24 October 2017 at 22:58, Stephen Boesch <java...@gmail.com> wrote: > Hi Mich, the github link has a brief intro - including a link to the > formal docs http://logisland.readthedocs.io/en/latest/index.html . They > have an architectural overview, developer guide, tutorial, and pretty > comprehensive api docs. > > 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > >> thanks Thomas. >> >> do you have a summary write-up for this tool please? >> >> >> regards, >> >> >> >> >> Thomas >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 24 October 2017 at 13:53, Thomas Bailet <thomas.bai...@hurence.com> >> wrote: >> >>> Hi >>> >>> we (@ hurence) have released on open source middleware based on >>> SparkStreaming over Kafka to do CEP and log mining, called *logisland* >>> (https://github.com/Hurence/logisland/) it has been deployed into >>> production for 2 years now and does a great job. You should have a look. >>> >>> >>> bye >>> >>> Thomas Bailet >>> >>> CTO : hurence >>> >>> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : >>> >>> As you may be aware the granularity that Spark streaming has is >>> micro-batching and that is limited to 0.5 second. So if you have continuous >>> ingestion of data then Spark streaming may not be granular enough for CEP. >>> You may consider other products. >>> >>> Worth looking at this old thread on mine "Spark support for Complex >>> Event Processing (CEP) >>> >>> https://mail-archives.apache.org/mod_mbox/spark-user/201604. >>> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@ >>> mail.gmail.com%3E >>> >>> HTH >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: >>> >>>> Hello all, >>>> >>>> Has anyone used spark streaming for CEP (Complex Event processing). >>>> Any CEP libraries that works well with spark. I have a use case for CEP and >>>> trying to see if spark streaming is a good fit. >>>> >>>> Currently we have a data pipeline using Kafka, Spark streaming and >>>> Cassandra for data ingestion and near real time dashboard. >>>> >>>> Please share your experience. >>>> Thanks much. >>>> -Anna >>>> >>>> >>>> >>> >>> >> >
Re: Spark streaming for CEP
Hi Mich, the github link has a brief intro - including a link to the formal docs http://logisland.readthedocs.io/en/latest/index.html . They have an architectural overview, developer guide, tutorial, and pretty comprehensive api docs. 2017-10-24 13:31 GMT-07:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > thanks Thomas. > > do you have a summary write-up for this tool please? > > > regards, > > > > > Thomas > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 24 October 2017 at 13:53, Thomas Bailet <thomas.bai...@hurence.com> > wrote: > >> Hi >> >> we (@ hurence) have released on open source middleware based on >> SparkStreaming over Kafka to do CEP and log mining, called *logisland* ( >> https://github.com/Hurence/logisland/) it has been deployed into >> production for 2 years now and does a great job. You should have a look. >> >> >> bye >> >> Thomas Bailet >> >> CTO : hurence >> >> Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : >> >> As you may be aware the granularity that Spark streaming has is >> micro-batching and that is limited to 0.5 second. So if you have continuous >> ingestion of data then Spark streaming may not be granular enough for CEP. >> You may consider other products. >> >> Worth looking at this old thread on mine "Spark support for Complex Event >> Processing (CEP) >> >> https://mail-archives.apache.org/mod_mbox/spark-user/201604. >> mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=LbQ@ >> mail.gmail.com%3E >> >> HTH >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: >> >>> Hello all, >>> >>> Has anyone used spark streaming for CEP (Complex Event processing). Any >>> CEP libraries that works well with spark. I have a use case for CEP and >>> trying to see if spark streaming is a good fit. >>> >>> Currently we have a data pipeline using Kafka, Spark streaming and >>> Cassandra for data ingestion and near real time dashboard. >>> >>> Please share your experience. >>> Thanks much. >>> -Anna >>> >>> >>> >> >> >
Re: Spark streaming for CEP
thanks Thomas. do you have a summary write-up for this tool please? regards, Thomas Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 24 October 2017 at 13:53, Thomas Bailet <thomas.bai...@hurence.com> wrote: > Hi > > we (@ hurence) have released on open source middleware based on > SparkStreaming over Kafka to do CEP and log mining, called *logisland* ( > https://github.com/Hurence/logisland/) it has been deployed into > production for 2 years now and does a great job. You should have a look. > > > bye > > Thomas Bailet > > CTO : hurence > > Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : > > As you may be aware the granularity that Spark streaming has is > micro-batching and that is limited to 0.5 second. So if you have continuous > ingestion of data then Spark streaming may not be granular enough for CEP. > You may consider other products. > > Worth looking at this old thread on mine "Spark support for Complex Event > Processing (CEP) > > https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/% > 3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=l...@mail.gmail.com%3E > > HTH > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: > >> Hello all, >> >> Has anyone used spark streaming for CEP (Complex Event processing). Any >> CEP libraries that works well with spark. I have a use case for CEP and >> trying to see if spark streaming is a good fit. >> >> Currently we have a data pipeline using Kafka, Spark streaming and >> Cassandra for data ingestion and near real time dashboard. >> >> Please share your experience. >> Thanks much. >> -Anna >> >> >> > >
Re: Spark streaming for CEP
Hi we (@ hurence) have released on open source middleware based on SparkStreaming over Kafka to do CEP and log mining, called *logisland* (https://github.com/Hurence/logisland/) it has been deployed into production for 2 years now and does a great job. You should have a look. bye Thomas Bailet CTO : hurence Le 18/10/17 à 22:05, Mich Talebzadeh a écrit : As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products. Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP) https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=l...@mail.gmail.com%3E HTH Dr Mich Talebzadeh LinkedIn /https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw/ http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk.Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com <mailto:annasta...@gmail.com>> wrote: Hello all, Has anyone used spark streaming for CEP (Complex Event processing). Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard. Please share your experience. Thanks much. -Anna
Re: Spark streaming for CEP
As you may be aware the granularity that Spark streaming has is micro-batching and that is limited to 0.5 second. So if you have continuous ingestion of data then Spark streaming may not be granular enough for CEP. You may consider other products. Worth looking at this old thread on mine "Spark support for Complex Event Processing (CEP) https://mail-archives.apache.org/mod_mbox/spark-user/201604.mbox/%3CCAJ3fcbB8eaf0JV84bA7XGUK5GajC1yGT3ZgTNCi8arJg56=l...@mail.gmail.com%3E HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 18 October 2017 at 20:52, anna stax <annasta...@gmail.com> wrote: > Hello all, > > Has anyone used spark streaming for CEP (Complex Event processing). Any > CEP libraries that works well with spark. I have a use case for CEP and > trying to see if spark streaming is a good fit. > > Currently we have a data pipeline using Kafka, Spark streaming and > Cassandra for data ingestion and near real time dashboard. > > Please share your experience. > Thanks much. > -Anna > > >
Spark streaming for CEP
Hello all, Has anyone used spark streaming for CEP (Complex Event processing). Any CEP libraries that works well with spark. I have a use case for CEP and trying to see if spark streaming is a good fit. Currently we have a data pipeline using Kafka, Spark streaming and Cassandra for data ingestion and near real time dashboard. Please share your experience. Thanks much. -Anna