RE Elastic Historically, we chose it for two reasons:
1. We wanted to discriminate ourselves with very rich data (also verbose). Lucene back-ends made sense in supporting a query language that would allow making full use of that data. 2. Kibana is great and greatly reduces the overhead in supporting front-end dashboard. It has a great feel to it, its highly customizable, and a plugin culture is growing. At the end of the day, absolutely nothing is stopping anyone from shipping logs to another back-end. I did get a ticket the other day to support a DAL… Best, Josh > On Jan 25, 2020, at 8:58 PM, Austin Bennett <[email protected]> > wrote: > > Great, thanks, Josh! (no worries on timing). > > A longer term nagging question, that still seems relevant -- Are there > any specific reasons that you've anchored on (what seems to only be) > Elastic as a backend? Naively, it seems could view Flagon as a way to > produce data, that then could go through a message queue and to one or > many different backends depending on desired consumption patterns. Is > this more a matter of convenience - Elastic suffices and therefore > devote attention elsewhere -- or because Elastic does something > particularly unique given the context of Flagon. > > Cheers, > Austin > > On Sat, Jan 25, 2020 at 5:15 PM Joshua Poore <[email protected]> wrote: >> >> Austin—good to hear from you. >> >> Apologies for the delay. Have been working with users last few days and >> wanted to give your email the appropriate attention—some good questions. >> >> Replies inline >> >>> On Jan 22, 2020, at 4:31 PM, Austin Bennett <[email protected]> >>> wrote: >>> >>> Hi Flagon, >>> >>> Looking for info: >>> >>> * I don't see much on analysing the data that gets collected. Very >>> interested in understanding the types of insights that people are >>> deriving. Pointers very welcome (esp. to specific use cases). >>> Apologies if missing something obvious… >> >> Most of our users are interested purely in business analytics. Mostly we get >> requests for guidance on Funnel’s and Heatmaps. We have a crude funnel >> mocked in a Kibana dashboard which you can find in our Business Analytics >> Dashboard here: >> https://github.com/apache/incubator-flagon/tree/master/docker/kibana/6.8.2/Saved%20Objects >> >> <https://github.com/apache/incubator-flagon/tree/master/docker/kibana/6.8.2/Saved%20Objects> >> >> Of course you’ll need to set up an Elastic instance to use it. You can find >> a sandbox ELK project in the parent directory: >> https://github.com/apache/incubator-flagon/tree/master/docker >> <https://github.com/apache/incubator-flagon/tree/master/docker> >> >> In the Kibana visualizations and dashboards you’ll find a number of other >> viz elements that derive directly from user asks. You’ll need to customize a >> little bit given you’ll have different values from your logs, but there is a >> LOT of content there. >> >> We are working on a python package, too, for more advanced behavioral >> analytics. We haven’t been able to devote much time to it as we’ve been >> working on tightening up UserALE, but we’ve done some WIP experiments with >> an analytic pipeline (seriously, very much a WIP): >> https://github.com/apache/incubator-flagon-distill/tree/distill_toolkit_refactor/WIP-distill_examples >> >> <https://github.com/apache/incubator-flagon-distill/tree/distill_toolkit_refactor/WIP-distill_examples> >>> >>> * Additionally, is JIRA the best place to understand pinpoints and >>> areas for improvement? Esp. upon recognizing value of using the data, >>> would be very interested in using my backend/data-eng experience to >>> help develop and increase the stability of system for high-scale >>> production use (sounds like successfully used places already). >> >> For now, yes, JIRA is the best place. We really do keep track of things well >> in JIRA. Most of the tickets in there are backlog ideas, wishes, task. We >> pull from the backlog into releases and track that way. In the very near >> term, we’ll be ditching JIRA and moving to Git Projects. JIRA seems to be a >> wall for our users. For now, feel free to jump in and add tickets labeled by >> component: >> >> For UserALE and log pipeline add tickets to: >> https://issues.apache.org/jira/browse/FLAGON-483?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20UserALE.js >> >> <https://issues.apache.org/jira/browse/FLAGON-483?jql=project%20=%20FLAGON%20AND%20component%20=%20UserALE.js> >> >> For Analytical asks, add tickets to: >> https://issues.apache.org/jira/browse/FLAGON-464?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20Distill >> >> <https://issues.apache.org/jira/browse/FLAGON-464?jql=project%20=%20FLAGON%20AND%20component%20=%20Distill> >> >> For Stack/back-end (ELK) improvements, add tickets to: >> https://issues.apache.org/jira/browse/FLAGON-466?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20stack >> >> <https://issues.apache.org/jira/browse/FLAGON-466?jql=project%20=%20FLAGON%20AND%20component%20=%20stack> >> >> Every commit we make, traces to a ticket. However, JIRA feels unwieldy for a >> lot of users. Feel free to add to JIRA, I will migrate to GIt Projects, when >> we make our move. >>> >>> * Lastly, how would y'all characterize the docs vs state of the code >>> (is it believed things are well documented and not much drift -- docs >>> tend to lag behind). The repository is not changing rapidly, so seems >>> like would be a slow drift if it has occurred. >> >> Lately, we’ve been burning down adoption issues for UserALE.js. We’re >> actually in the final throes of testing and documenting for UserALE.js v >> 2.1.0. This is actually a largish release: >> >> 1. Webpack/Module Bundler support >> 2. SessionStorage Support >> 3. custom auth headers >> 4. comprehensive custom log support >> 5. other things >> >> We’ve been pretty active there: >> https://github.com/apache/incubator-flagon-useralejs/tree/FLAGON-469 >> <https://github.com/apache/incubator-flagon-useralejs/tree/FLAGON-469> >> >> As per docs, our workflow generally follows a pattern such that all READMEs >> are updated prior to pushing new release candidates. Once released, we >> update docs on the website at the same time that we update our website to >> reflect new release links, etc. >> >> Currently, docs on UserALE.js — >> http://flagon.incubator.apache.org/docs/useralejs/ >> <http://flagon.incubator.apache.org/docs/useralejs/> — and the stack — >> http://flagon.incubator.apache.org/docs/stack/ >> <http://flagon.incubator.apache.org/docs/stack/> — are about up to date, or >> lag by a patch release. The website is generally a good source for guides >> and considerations, but our readmes are pretty solid. We’re always willing >> to field asks for more specific documentation. >> >> We would love some help on our back-end. We’re lagging a bit as we’ve not >> yet moved our examples up from ELK 6.8—>7.0. That’s on the near term slate, >> as well as more documentation on clustering and indexing options. But, we’d >> love PRs. >> >>> >>> Thanks, >>> Austin >> >> Thank You, Austin! I hope this is helpful! >>
