Thanks for the feedback everyone. Keith, we are currently exporting to Kafka, but I don't foresee needing to ingest from Kafka into Fluo for this project. I will certainly keep you updated if that changes.
Thanks, Caleb A. Meier, Ph.D. Software Engineer II ♦ Analyst Parsons Corporation 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209 Office: (703)797-3066 [email protected] ♦ www.parsons.com -----Original Message----- From: Keith Turner [mailto:[email protected]] Sent: Friday, February 03, 2017 10:05 AM To: [email protected] Subject: Re: third party service to poll Fluo for absence of event If you have time to share as you move forward, I am interested in learning from your experiences with using Kafka and Fluo together. I have wanted to experiment with this in order to see if Fluo needed any changes to support better interoperation. However, I have not had the time. I opened #795[1], but its based on speculation not experience. [1]: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dfluo_issues_795&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=5UoPZQJ9JMRl_7xZ5RY_Yw7cEUFMTQQLpbr341mlILw&s=fcHGc-fgm-SIG8XLgsQyvHOlpRVexhcGQWSlCIQETtA&e= On Thu, Feb 2, 2017 at 5:15 PM, Meier, Caleb <[email protected]> wrote: > Thanks for the input. I'm currently looking at creating some sort of > coordinator (which wraps a ScheduledExecutorService to generate periodic > notifications) and a collection of workers (to process the periodic queries > as they are issued). Most of the interaction between the workers and > coordinator will be via Kafka (develop some sort of protocol to ensure that > more than one worker isn't getting assigned the same query). At any rate, I > was thinking of implementing these components as TwillRunnables. However, it > seems like the Twill documentation is a bit sparse. Given that you guys > implemented Fluo as a TwillApplication, do you have any insight/advice for > writing TwillApplications? In particular, how is your FluoTwillApp being > run? All of the examples I've seen create a client with a TwillRunner and > TwillController. It seems like you 've created your own version of a > YarnAppRunner -- what role is that playing in running the FluoTwillApp? > Moreover, it is also unclear to me whether the TwillRunnables are bound to > the client -- if the client terminates do the runnables terminate as well? > So essentially, it is unclear to me how create a long running application in > Twill that is not bound to a particular client. Sorry that this is a little > off topic, but any help, references to documentation/examples would be very > appreciated. > > Caleb A. Meier, Ph.D. > Software Engineer II ♦ Analyst > Parsons Corporation > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209 > Office: (703)797-3066 > [email protected] ♦ www.parsons.com > > -----Original Message----- > From: Keith Turner [mailto:[email protected]] > Sent: Wednesday, February 01, 2017 11:03 PM > To: [email protected] > Subject: Re: third party service to poll Fluo for absence of event > > On Wed, Feb 1, 2017 at 9:54 PM, Christopher <[email protected]> wrote: >> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb >> <[email protected]> >> wrote: >> >>> Yeah, this seems pretty reasonable to me. I guess it then boils >>> down to the nitty gritty of do I store results in Fluo and have my >>> service query Fluo (I think you guys actually advise against that in >>> your documentation), or export results and then have the service >>> query some external index that I am exporting to. >>> >>> >> I'm not sure we advise against it, so much as recognize that it may >> not be suitable for certain use cases and may not meet query >> performance expectations ( >> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_&d=CwIFaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk&s=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs&e= >> ). >> > > I would advise against querying Fluo for low latency queries. > However, this external service thats checking a few stats within Fluo and > injecting new notifications probably does not care about latency. > > The reason Fluo is not geared towards low latency is that it does lazy > recovery of failed transactions. Failed transactions are not cleaned > up until something tries to read the data, which could significantly delay > reads. > >> In any case, your observer need not write the final "last occurrence" >> entries into a Fluo table. It could write them anywhere. >> >> >>> Regarding timestamps, does the oracle server provide actual >>> timestamps or just logical timestamps? That is, could I use the >>> timestamps that the server provides to define some sort of now() >>> function to obtain the current time to compare with the times of incoming >>> events? >>> >> >> Just logical time, and it delivers batches to limit locking, so it >> can appear to jump ahead spontaneously. I'm not sure the OracleServer >> is suitable for this purpose. What level of precision are you going for? >> It might be enough to just run NTP, if you don't need more precision >> than "within seconds". >> >> >>> ________________________________________ >>> From: Christopher <[email protected]> >>> Sent: Tuesday, January 31, 2017 5:08 PM >>> To: [email protected] >>> Subject: Re: third party service to poll Fluo for absence of event >>> >>> You could write an observer which rolls up timestamps from all the >>> events you are concerned about, and puts the most recent event >>> timestamp into a centralized place, which you could poll. If there >>> is no ingest of these events, then the last timestamp in this >>> central place will exceed some threshold and the poller could detect that >>> and trigger additional actions. >>> >>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb >>> <[email protected]> >>> wrote: >>> >>> > Hello, >>> > >>> > I’m looking into using Fluo to develop an event based notification >>> > system that incrementally generates events of increasing >>> > complexity. The one issue that I’m running into is how to handle >>> > the non-event event. That >>> is, >>> > Fluo (as I understand it) is not well-suited to handle the >>> > following >>> > request: “generate a notification if no events of a given type >>> > have occurred within the last 24 hours”. This is because it is a >>> > push based notification framework that only generates >>> > notifications when things actually happen. So the question is, >>> > has anyone looked into developing a service for generating >>> > notifications at regular intervals (even if something doesn’t happen) >>> > that works with Fluo? >>> > I’m toying with the idea of creating some sort of Twill >>> > application that tells Fluo to wake up at regular intervals to >>> > generate a notification about the set of events falling within the >>> > given time window. Before doing this I just wanted to make sure >>> > that something like this does not already exist, and I also >>> want >>> > to get a sense of how bad an idea it is to delegate some of the logic of >>> > this periodic notification service to Fluo. Would it be better to >>> > separate out the temporal portion of my notification request to be >>> > processed entirely outside of Fluo to avoid transactional overhead? >>> > >>> > Caleb A. Meier, Ph.D. >>> > Software Engineer II ♦ Analyst >>> > Parsons Corporation >>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209 >>> > Office: (703)797-3066 <(703)%20797-3066> <(703)%20797-3066> >>> > [email protected]<mailto:[email protected]> ♦ >>> www.parsons.com< >>> > http://www.parsons.com/> >>> > >>> > -- >>> Christopher >>> >> -- >> Christopher
