Re: Spark support for Complex Event Processing (CEP)

Sorry for answering delay.. Yes, this is not pure "CEP", but quite close for it or many similar "functionalities". My case is not so easy, because i dont' want to compare against original time schedule of route. I want to compare how close (ITS) system has estimated arrival time to bus

Re: Spark support for Complex Event Processing (CEP)

If you’re getting the logs, then it really isn’t CEP unless you consider the event to be the log from the bus. This doesn’t sound like there is a time constraint. Your bus schedule is fairly fixed and changes infrequently. Your bus stops are relatively fixed points. (Within a couple of

Re: Spark support for Complex Event Processing (CEP)

ok like any work you need to start this from a simple model. take one bus only (identified by bus number which is unique). for any bus no N you have two logs LOG A and LOG B and LOG C the coordinator from Central computer that sends estimated time of arrival (ETA) to the bus stops. Pretty simple.

Re: Spark support for Complex Event Processing (CEP)

Hi I try to explain my case .. Situation is not so simple in my logs and solution. There also many types of logs and there are from many sources. They are as csv-format and header line includes names of the columns. This is simplified description of input logs. LOG A's: bus coordinate logs

Re: Spark support for Complex Event Processing (CEP)

Look, you said that you didn’t have continuous data, and you do have continuous data. I just used an analog signal which can be converted. So that you end up with contiguous digital sampling. The point is that you have to consider that micro batches are still batched and you’re adding

Re: Spark support for Complex Event Processing (CEP)

Also the point about "First there is this thing called analog signal processing…. Is that continuous enough for you? " I agree that analog signal processing like a sine wave, an AM radio signal – is truly continuous. However, here we are talking about digital data which will always be sent as

Re: Spark support for Complex Event Processing (CEP)

In a commerical (C)EP like say StreamBase, or for example its competitor Apama, the arrival of an input event **immediately** triggers further downstream processing. This is admitadly an asynchronous approach, not a synchronous clock-driven micro-batch approach like Spark's. I suppose if one

Re: Spark support for Complex Event Processing (CEP)

I don’t. I believe that there have been a couple of hack-a-thons like one done in Chicago a few years back using public transportation data. The first question is what sort of data do you get from the city? I mean it could be as simple as time_stamp, bus_id, route and GPS (x,y). Or they

Re: Spark support for Complex Event Processing (CEP)

Do you know any good examples how to use Spark streaming in tracking public transportation systems ? Or Storm or some other tool example ? Regards Esa Heikkinen 28.4.2016, 3:16, Michael Segel kirjoitti: Uhm… I think you need to clarify a couple of things… First there is this thing called

Fwd: Spark support for Complex Event Processing (CEP)

Doh! Wrong email account again! > Begin forwarded message: > > From: Michael Segel <michael_se...@hotmail.com> > Subject: Re: Spark support for Complex Event Processing (CEP) > Date: April 27, 2016 at 7:16:55 PM CDT > To: Mich Talebzadeh <mich.talebza...@gma

Re: Spark support for Complex Event Processing (CEP)

couple of things. There is no such thing as Continuous Data Streaming as there is no such thing as Continuous Availability. There is such thing as Discrete Data Streaming and High Availability but they reduce the finite unavailability to minimum. In terms of business needs a 5 SIGMA is good

Re: Spark support for Complex Event Processing (CEP)

Hi Thanks for the answer. I have developed a log file analyzer for RTPIS (Real Time Passenger Information System) system, where buses drive lines and the system try to estimate the arrival times to the bus stops. There are many different log files (and events) and analyzing situation can be

Fwd: Spark support for Complex Event Processing (CEP)

Sorry sent from wrong email address. > Begin forwarded message: > > From: Michael Segel <michael_se...@hotmail.com> > Subject: Re: Spark support for Complex Event Processing (CEP) > Date: April 27, 2016 at 7:51:14 AM CDT > To: Mich Talebzadeh <mich.talebza...@gma

Re: Spark support for Complex Event Processing (CEP)

please see my other reply Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 27 April 2016 at 10:40, Esa

Re: Spark support for Complex Event Processing (CEP)

Hi I have followed with interest the discussion about CEP and Spark. It is quite close to my research, which is a complex analyzing for log files and "history" data (not actually for real time streams). I have few questions: 1) Is CEP only for (real time) stream data and not for "history"

Re: Spark support for Complex Event Processing (CEP)

cision> > >From: Alonso Isidoro Roman <*alons...@gmail.com* <alons...@gmail.com>> >To: Mich Talebzadeh <*mich.talebza...@gmail.com* ><mich.talebza...@gmail.com>> >Cc: Mario Ds Briggs/India/IBM@IBMIN, Luciano Resende < >*luckbr1...@gmail.c

Re: Spark support for Complex Event Processing (CEP)

4/2016 08:34 pm Subject: Re: Spark support for Complex Event Processing (CEP) Hi, Following example in https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532 Does anyone know which jar file this belon

Re: Spark support for Complex Event Processing (CEP)

za...@gmail.com> > Cc: Mario Ds Briggs/India/IBM@IBMIN, Luciano Resende <luckbr1...@gmail.com>, > "user @spark" <user@spark.apache.org> > Date: 20/04/2016 02:24 pm > Subject: Re: Spark support for Complex Event Processing (CEP) >

Re: Spark support for Complex Event Processing (CEP)

>, "user @spark" <user@spark.apache.org> Date: 20/04/2016 02:24 pm Subject: Re: Spark support for Complex Event Processing (CEP) Stratio decision could do the job https://github.com/Stratio/Decision Alonso Isidoro Roman. Mis citas preferidas (de hoy) : "Si depu

Re: Spark support for Complex Event Processing (CEP)

Briggs/India/IBM@IBMIN > Cc: "user @spark" <user@spark.apache.org>, Luciano Resende < > luckbr1...@gmail.com> > Date: 19/04/2016 12:45 am > Subject: Re: Spark support for Complex Event Processing (CEP) > -- > > &g

Re: Spark support for Complex Event Processing (CEP)

g>, Luciano Resende <luckbr1...@gmail.com> Date: 19/04/2016 12:45 am Subject: Re: Spark support for Complex Event Processing (CEP) great stuff Mario. Much appreciated. Mich Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianr

Re: Spark support for Complex Event Processing (CEP)

00 - > > *To:* > Luciano Resende <luckbr1...@gmail.com> > > *cc:* > "user @spark" <user@spark.apache.org> > > *Subject:* > Re: Spark support for Complex Event Processing (CEP)Thanks Luciano. > Appreciated. > > Regards > > Dr Mich Talebzad

Re: Spark support for Complex Event Processing (CEP)

gmail.com> cc: "user @spark" <user@spark.apache.org> Subject: Re: Spark support for Complex Event

Re: Spark support for Complex Event Processing (CEP)

Thanks Luciano. Appreciated. Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 17 April 2016

Re: Spark support for Complex Event Processing (CEP)

Hi Mitch, I know some folks that were investigating/prototyping on this area, let me see if I can get them to reply here with more details. On Sun, Apr 17, 2016 at 1:54 AM, Mich Talebzadeh wrote: > Hi, > > Has Spark got libraries for CEP using Spark Streaming with

Spark support for Complex Event Processing (CEP)

Hi, Has Spark got libraries for CEP using Spark Streaming with Kafka by any chance? I am looking at Flink that supposed to have these libraries for CEP but I find Flink itself very much work in progress. Thanks Dr Mich Talebzadeh LinkedIn *