Re: third party service to poll Fluo for absence of event

[Keith Turner]

On Wed, Feb 1, 2017 at 9:54 PM, Christopher  wrote:
> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb 
> wrote:
>
>> Yeah, this seems pretty reasonable to me.  I guess it then boils down to
>> the nitty gritty of do I store results in Fluo and have my service query
>> Fluo (I think you guys actually advise against that in your documentation),
>> or export results and then have the service query some external index that
>> I am exporting to.
>>
>>
> I'm not sure we advise against it, so much as recognize that it may not be
> suitable for certain use cases and may not meet query performance
> expectations (
> http://fluo.apache.org/docs/fluo-recipes/1.0.0-incubating/export-queue/).
>

I would advise against querying Fluo for low latency queries.
However, this external service thats checking a few stats within Fluo
and injecting new notifications probably does not care about latency.

The reason Fluo is not geared towards low latency is that it does lazy
recovery of failed transactions.   Failed transactions are not cleaned
up until something tries to read the data, which could significantly
delay reads.

> In any case, your observer need not write the final "last occurrence"
> entries into a Fluo table. It could write them anywhere.
>
>
>> Regarding timestamps, does the oracle server provide actual timestamps or
>> just logical timestamps?  That is, could I use the timestamps that the
>> server provides to define some sort of now() function to obtain the current
>> time to compare with the times of incoming events?
>>
>
> Just logical time, and it delivers batches to limit locking, so it can
> appear to jump ahead spontaneously. I'm not sure the OracleServer is
> suitable for this purpose. What level of precision are you going for? It
> might be enough to just run NTP, if you don't need more precision than
> "within seconds".
>
>
>> 
>> From: Christopher 
>> Sent: Tuesday, January 31, 2017 5:08 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: third party service to poll Fluo for absence of event
>>
>> You could write an observer which rolls up timestamps from all the events
>> you are concerned about, and puts the most recent event timestamp into a
>> centralized place, which you could poll. If there is no ingest of these
>> events, then the last timestamp in this central place will exceed some
>> threshold and the poller could detect that and trigger additional actions.
>>
>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb 
>> wrote:
>>
>> > Hello,
>> >
>> > I’m looking into using Fluo to develop an event based notification system
>> > that incrementally generates events of increasing complexity.  The one
>> > issue that I’m running into is how to handle the non-event event.  That
>> is,
>> > Fluo (as I understand it) is not well-suited to handle the following
>> > request: “generate a notification if no events of a given type have
>> > occurred within the last 24 hours”.  This is because it is a push based
>> > notification framework that only generates notifications when things
>> > actually happen.  So the question is, has anyone looked into developing a
>> > service for generating notifications at regular intervals (even if
>> > something doesn’t happen) that works with Fluo?  I’m toying with the idea
>> > of creating some sort of Twill application that tells Fluo to wake up at
>> > regular intervals to generate a notification about the set of events
>> > falling within the given time window. Before doing this I just wanted to
>> > make sure that something like this does not already exist, and I also
>> want
>> > to get a sense of how bad an idea it is to delegate some of the logic of
>> > this periodic notification service to Fluo.   Would it be better to
>> > separate out the temporal portion of my notification request to be
>> > processed entirely outside of Fluo to avoid transactional overhead?
>> >
>> > Caleb A. Meier, Ph.D.
>> > Software Engineer II ♦ Analyst
>> > Parsons Corporation
>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> > Office:  (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>> > caleb.me...@parsons.com ♦
>> www.parsons.com<
>> > http://www.parsons.com/>
>> >
>> > --
>> Christopher
>>
> --
> Christopher

Re: third party service to poll Fluo for absence of event

[Keith Turner]

On Wed, Feb 1, 2017 at 9:53 AM, Meier, Caleb  wrote:
> Hey Keith,
>
> Right now the project is in its infancy so I don't have concrete examples of 
> how events are distributed, but I think we would like everything to scale as 
> needed.  Right now, the user is in charge of how events are modeled and how 
> many different event types are included.
>
> Regarding the use cases that you mentioned, I think we would be more 
> interested in find which event types haven't been updated or haven't occurred 
> within the last 24 hours.  I'm not ruling out the possibility of tracking 
> updates to an individual event though.
>
> More generally, we're interested in issuing periodic notifications.  The 
> non-event event is kind of a canonical example.  Another use case would be 
> "generate a notification if more than 100 events of a given type have 
> occurred within the last 24 hours",  and the user specifies that they want a 
> notification to be generated every hour.  In the case that 1000 events happen 
> in the first hour and then no other events occur during that 24 hour period, 
> 1000 notifications would be generated by Fluo during the first hour, but then 
> nothing would happen the remaining 23 hours.  We would need some service 
> (that possibly utilizes an observer if that makes sense) to periodically 
> detect how many events have occurred within the specified window and generate 
> a notification.  So in this case, every hour for the next 23 hours, something 
> would check and find that 1000 events occurred within the window and generate 
> a notification.
>
> Did that clarify things?

Yeah.  I feel like this could be broken into two parts.

 * A many to few update recipe with short circuiting.
 * Something that periodically scans the "few" row/cols and possibly
injects a new action.

Below is an example to show what I mean by short circuiting.

 1. Processing event 100934 with event type X
 2. Update the last update hour for event type X only if its value is
less than current hour

The Collision Free Map (CFM) could be used for these many to few
updates.  However, it lacks the short circuit optimization.  It will
always queue the updates, even when the applied update will not change
the current value.

The external scan service periodically scan the few event type last
update times and inject new notifications.  This mechanism could use
transactions to avoid problems from accidentally running multiple
external services.

> 
> From: Keith Turner 
> Sent: Tuesday, January 31, 2017 6:43 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb  wrote:
>> Hello,
>>
>> I’m looking into using Fluo to develop an event based notification system 
>> that incrementally generates events of increasing complexity.  The one issue 
>> that I’m running into is how to handle the non-event event.  That is, Fluo 
>> (as I understand it) is not well-suited to handle the following request: 
>> “generate a notification if no events of a given type have occurred within 
>> the last 24 hours”.  This is because it is a push based notification 
>> framework that only generates notifications when things actually happen.  So 
>> the question is, has anyone looked into developing a service for generating 
>> notifications at regular intervals (even if something doesn’t happen) that 
>> works with Fluo?  I’m toying with the idea of creating some sort of Twill 
>> application that tells Fluo to wake up at regular intervals to generate a 
>> notification about the set of events falling within the given time window. 
>> Before doing this I just wanted to make sure that something like this does 
>> not already exist, and I also want to get a sense of how bad an idea it is 
>> to delegate some of the logic of this periodic notification service to Fluo. 
>>   Would it be better to separate out the temporal portion of my notification 
>> request to be processed entirely outside of Fluo to avoid transactional 
>> overhead?
>>
>
> At one point in time I was thinking of delayed notifications, but I
> can't remember what my specific use case was. I am not sure if this is
> good idea or not.
>
> How many event types are there in your use case?  Just curious about
> the order of magnitude.  Like is it ~100 event types and millions of
> events?
>
> Just to make sure I understand your use case, are you interested in
> finding which event types were not updated in the last 24hrs?  Or are
> you interested in finding which events were not updated in the last
> 24hrs?  If none of these, would you be able to describe the use case
> you want with an example?
>
>> Caleb A. Meier, Ph.D.
>> Software Engineer II ♦ Analyst
>> Parsons Corporation
>> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> Office:  (703)797-3066
>> 

Re: third party service to poll Fluo for absence of event

[Christopher]

On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb 
wrote:

> Yeah, this seems pretty reasonable to me.  I guess it then boils down to
> the nitty gritty of do I store results in Fluo and have my service query
> Fluo (I think you guys actually advise against that in your documentation),
> or export results and then have the service query some external index that
> I am exporting to.
>
>
I'm not sure we advise against it, so much as recognize that it may not be
suitable for certain use cases and may not meet query performance
expectations (
http://fluo.apache.org/docs/fluo-recipes/1.0.0-incubating/export-queue/).

In any case, your observer need not write the final "last occurrence"
entries into a Fluo table. It could write them anywhere.


> Regarding timestamps, does the oracle server provide actual timestamps or
> just logical timestamps?  That is, could I use the timestamps that the
> server provides to define some sort of now() function to obtain the current
> time to compare with the times of incoming events?
>

Just logical time, and it delivers batches to limit locking, so it can
appear to jump ahead spontaneously. I'm not sure the OracleServer is
suitable for this purpose. What level of precision are you going for? It
might be enough to just run NTP, if you don't need more precision than
"within seconds".


> 
> From: Christopher 
> Sent: Tuesday, January 31, 2017 5:08 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> You could write an observer which rolls up timestamps from all the events
> you are concerned about, and puts the most recent event timestamp into a
> centralized place, which you could poll. If there is no ingest of these
> events, then the last timestamp in this central place will exceed some
> threshold and the poller could detect that and trigger additional actions.
>
> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb 
> wrote:
>
> > Hello,
> >
> > I’m looking into using Fluo to develop an event based notification system
> > that incrementally generates events of increasing complexity.  The one
> > issue that I’m running into is how to handle the non-event event.  That
> is,
> > Fluo (as I understand it) is not well-suited to handle the following
> > request: “generate a notification if no events of a given type have
> > occurred within the last 24 hours”.  This is because it is a push based
> > notification framework that only generates notifications when things
> > actually happen.  So the question is, has anyone looked into developing a
> > service for generating notifications at regular intervals (even if
> > something doesn’t happen) that works with Fluo?  I’m toying with the idea
> > of creating some sort of Twill application that tells Fluo to wake up at
> > regular intervals to generate a notification about the set of events
> > falling within the given time window. Before doing this I just wanted to
> > make sure that something like this does not already exist, and I also
> want
> > to get a sense of how bad an idea it is to delegate some of the logic of
> > this periodic notification service to Fluo.   Would it be better to
> > separate out the temporal portion of my notification request to be
> > processed entirely outside of Fluo to avoid transactional overhead?
> >
> > Caleb A. Meier, Ph.D.
> > Software Engineer II ♦ Analyst
> > Parsons Corporation
> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> > Office:  (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
> > caleb.me...@parsons.com ♦
> www.parsons.com<
> > http://www.parsons.com/>
> >
> > --
> Christopher
>
-- 
Christopher

Podling Report Reminder - February 2017

[johndament]

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, February 01).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.

This should be appended to the Incubator Wiki page at:

https://wiki.apache.org/incubator/February2017

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: Podling Report Reminder - February 2017

[Christopher]

+1

I was hoping to start a graduation discussion at some point, and thought
that make it into a report, but maybe next time.

In particular, I don't know that there is much left for us to learn in
incubation. The biggest issue is growth... and I think incubation can only
go so far in helping with that. It might be worth discussing graduation,
with a potential landing point as a sub-project of the Accumulo PMC, if
they are willing to support us as a sub-project. In that way, we could
inherit a larger voting base for basic release verification, in case that's
a risk of us being on our own. I don't think there's a serious risk of the
project dying... but there might be a serious risk of us failing to meet
TLP expectations for growth. Every project grows at a different pace, and
Fluo just might happen to be on the very slow end of that scale.

On Wed, Feb 1, 2017 at 11:21 AM Mike Walch  wrote:

> Looks great.  Thanks Keith!
>
> On Wed, Feb 1, 2017 at 10:48 AM Keith Turner  wrote:
>
> > I updated the Fluo report on the wiki.  Does anyone have any suggested
> > changes?
> >
> > https://wiki.apache.org/incubator/February2017
> >
> > On Tue, Jan 31, 2017 at 10:02 PM,   wrote:
> > > Dear podling,
> > >
> > > This email was sent by an automated system on behalf of the Apache
> > > Incubator PMC. It is an initial reminder to give you plenty of time to
> > > prepare your quarterly board report.
> > >
> > > The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
> > > The report for your podling will form a part of the Incubator PMC
> > > report. The Incubator PMC requires your report to be submitted 2 weeks
> > > before the board meeting, to allow sufficient time for review and
> > > submission (Wed, February 01).
> > >
> > > Please submit your report with sufficient time to allow the Incubator
> > > PMC, and subsequently board members to review and digest. Again, the
> > > very latest you should submit your report is 2 weeks prior to the board
> > > meeting.
> > >
> > > Thanks,
> > >
> > > The Apache Incubator PMC
> > >
> > > Submitting your Report
> > >
> > > --
> > >
> > > Your report should contain the following:
> > >
> > > *   Your project name
> > > *   A brief description of your project, which assumes no knowledge of
> > > the project or necessarily of its field
> > > *   A list of the three most important issues to address in the move
> > > towards graduation.
> > > *   Any issues that the Incubator PMC or ASF Board might wish/need to
> be
> > > aware of
> > > *   How has the community developed since the last report
> > > *   How has the project developed since the last report.
> > >
> > > This should be appended to the Incubator Wiki page at:
> > >
> > > https://wiki.apache.org/incubator/February2017
> > >
> > > Note: This is manually populated. You may need to wait a little before
> > > this page is created from a template.
> > >
> > > Mentors
> > > ---
> > >
> > > Mentors should review reports for their project(s) and sign them off on
> > > the Incubator wiki page. Signing off reports shows that you are
> > > following the project - projects that are not signed may raise alarms
> > > for the Incubator PMC.
> > >
> > > Incubator PMC
> >
>
-- 
Christopher

Re: Podling Report Reminder - February 2017

[Mike Walch]

Looks great.  Thanks Keith!

On Wed, Feb 1, 2017 at 10:48 AM Keith Turner  wrote:

> I updated the Fluo report on the wiki.  Does anyone have any suggested
> changes?
>
> https://wiki.apache.org/incubator/February2017
>
> On Tue, Jan 31, 2017 at 10:02 PM,   wrote:
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, February 01).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://wiki.apache.org/incubator/February2017
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
>

Re: Podling Report Reminder - February 2017

[Keith Turner]

I updated the Fluo report on the wiki.  Does anyone have any suggested changes?

https://wiki.apache.org/incubator/February2017

On Tue, Jan 31, 2017 at 10:02 PM,   wrote:
> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, February 01).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/February2017
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC

Re: third party service to poll Fluo for absence of event

[Meier, Caleb]

Yeah, this seems pretty reasonable to me.  I guess it then boils down to the 
nitty gritty of do I store results in Fluo and have my service query Fluo (I 
think you guys actually advise against that in your documentation), or export 
results and then have the service query some external index that I am exporting 
to.  

Regarding timestamps, does the oracle server provide actual timestamps or just 
logical timestamps?  That is, could I use the timestamps that the server 
provides to define some sort of now() function to obtain the current time to 
compare with the times of incoming events?

From: Christopher 
Sent: Tuesday, January 31, 2017 5:08 PM
To: dev@fluo.incubator.apache.org
Subject: Re: third party service to poll Fluo for absence of event

You could write an observer which rolls up timestamps from all the events
you are concerned about, and puts the most recent event timestamp into a
centralized place, which you could poll. If there is no ingest of these
events, then the last timestamp in this central place will exceed some
threshold and the poller could detect that and trigger additional actions.

On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb 
wrote:

> Hello,
>
> I’m looking into using Fluo to develop an event based notification system
> that incrementally generates events of increasing complexity.  The one
> issue that I’m running into is how to handle the non-event event.  That is,
> Fluo (as I understand it) is not well-suited to handle the following
> request: “generate a notification if no events of a given type have
> occurred within the last 24 hours”.  This is because it is a push based
> notification framework that only generates notifications when things
> actually happen.  So the question is, has anyone looked into developing a
> service for generating notifications at regular intervals (even if
> something doesn’t happen) that works with Fluo?  I’m toying with the idea
> of creating some sort of Twill application that tells Fluo to wake up at
> regular intervals to generate a notification about the set of events
> falling within the given time window. Before doing this I just wanted to
> make sure that something like this does not already exist, and I also want
> to get a sense of how bad an idea it is to delegate some of the logic of
> this periodic notification service to Fluo.   Would it be better to
> separate out the temporal portion of my notification request to be
> processed entirely outside of Fluo to avoid transactional overhead?
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office:  (703)797-3066 <(703)%20797-3066>
> caleb.me...@parsons.com ♦ www.parsons.com<
> http://www.parsons.com/>
>
> --
Christopher