That's what I am hoping to understand -- making full use of the data :-) It certainly goes beyond business analytics, funnels, heatmaps.
On Sun, Jan 26, 2020, 8:47 PM Joshua Poore <[email protected]> wrote: > RE Elastic > > Historically, we chose it for two reasons: > > 1. We wanted to discriminate ourselves with very rich data (also verbose). > Lucene back-ends made sense in supporting a query language that would allow > making full use of that data. > > 2. Kibana is great and greatly reduces the overhead in supporting > front-end dashboard. It has a great feel to it, its highly customizable, > and a plugin culture is growing. > > At the end of the day, absolutely nothing is stopping anyone from shipping > logs to another back-end. I did get a ticket the other day to support a DAL… > > Best, > > Josh > > > On Jan 25, 2020, at 8:58 PM, Austin Bennett <[email protected]> > wrote: > > > > Great, thanks, Josh! (no worries on timing). > > > > A longer term nagging question, that still seems relevant -- Are there > > any specific reasons that you've anchored on (what seems to only be) > > Elastic as a backend? Naively, it seems could view Flagon as a way to > > produce data, that then could go through a message queue and to one or > > many different backends depending on desired consumption patterns. Is > > this more a matter of convenience - Elastic suffices and therefore > > devote attention elsewhere -- or because Elastic does something > > particularly unique given the context of Flagon. > > > > Cheers, > > Austin > > > > On Sat, Jan 25, 2020 at 5:15 PM Joshua Poore <[email protected]> wrote: > >> > >> Austin—good to hear from you. > >> > >> Apologies for the delay. Have been working with users last few days and > wanted to give your email the appropriate attention—some good questions. > >> > >> Replies inline > >> > >>> On Jan 22, 2020, at 4:31 PM, Austin Bennett < > [email protected]> wrote: > >>> > >>> Hi Flagon, > >>> > >>> Looking for info: > >>> > >>> * I don't see much on analysing the data that gets collected. Very > >>> interested in understanding the types of insights that people are > >>> deriving. Pointers very welcome (esp. to specific use cases). > >>> Apologies if missing something obvious… > >> > >> Most of our users are interested purely in business analytics. Mostly > we get requests for guidance on Funnel’s and Heatmaps. We have a crude > funnel mocked in a Kibana dashboard which you can find in our Business > Analytics Dashboard here: > https://github.com/apache/incubator-flagon/tree/master/docker/kibana/6.8.2/Saved%20Objects > < > https://github.com/apache/incubator-flagon/tree/master/docker/kibana/6.8.2/Saved%20Objects > > > >> > >> Of course you’ll need to set up an Elastic instance to use it. You can > find a sandbox ELK project in the parent directory: > https://github.com/apache/incubator-flagon/tree/master/docker < > https://github.com/apache/incubator-flagon/tree/master/docker> > >> > >> In the Kibana visualizations and dashboards you’ll find a number of > other viz elements that derive directly from user asks. You’ll need to > customize a little bit given you’ll have different values from your logs, > but there is a LOT of content there. > >> > >> We are working on a python package, too, for more advanced behavioral > analytics. We haven’t been able to devote much time to it as we’ve been > working on tightening up UserALE, but we’ve done some WIP experiments with > an analytic pipeline (seriously, very much a WIP): > https://github.com/apache/incubator-flagon-distill/tree/distill_toolkit_refactor/WIP-distill_examples > < > https://github.com/apache/incubator-flagon-distill/tree/distill_toolkit_refactor/WIP-distill_examples > > > >>> > >>> * Additionally, is JIRA the best place to understand pinpoints and > >>> areas for improvement? Esp. upon recognizing value of using the data, > >>> would be very interested in using my backend/data-eng experience to > >>> help develop and increase the stability of system for high-scale > >>> production use (sounds like successfully used places already). > >> > >> For now, yes, JIRA is the best place. We really do keep track of things > well in JIRA. Most of the tickets in there are backlog ideas, wishes, task. > We pull from the backlog into releases and track that way. In the very near > term, we’ll be ditching JIRA and moving to Git Projects. JIRA seems to be a > wall for our users. For now, feel free to jump in and add tickets labeled > by component: > >> > >> For UserALE and log pipeline add tickets to: > https://issues.apache.org/jira/browse/FLAGON-483?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20UserALE.js > < > https://issues.apache.org/jira/browse/FLAGON-483?jql=project%20=%20FLAGON%20AND%20component%20=%20UserALE.js > > > >> > >> For Analytical asks, add tickets to: > https://issues.apache.org/jira/browse/FLAGON-464?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20Distill > < > https://issues.apache.org/jira/browse/FLAGON-464?jql=project%20=%20FLAGON%20AND%20component%20=%20Distill > > > >> > >> For Stack/back-end (ELK) improvements, add tickets to: > https://issues.apache.org/jira/browse/FLAGON-466?jql=project%20%3D%20FLAGON%20AND%20component%20%3D%20stack > < > https://issues.apache.org/jira/browse/FLAGON-466?jql=project%20=%20FLAGON%20AND%20component%20=%20stack > > > >> > >> Every commit we make, traces to a ticket. However, JIRA feels unwieldy > for a lot of users. Feel free to add to JIRA, I will migrate to GIt > Projects, when we make our move. > >>> > >>> * Lastly, how would y'all characterize the docs vs state of the code > >>> (is it believed things are well documented and not much drift -- docs > >>> tend to lag behind). The repository is not changing rapidly, so seems > >>> like would be a slow drift if it has occurred. > >> > >> Lately, we’ve been burning down adoption issues for UserALE.js. We’re > actually in the final throes of testing and documenting for UserALE.js v > 2.1.0. This is actually a largish release: > >> > >> 1. Webpack/Module Bundler support > >> 2. SessionStorage Support > >> 3. custom auth headers > >> 4. comprehensive custom log support > >> 5. other things > >> > >> We’ve been pretty active there: > https://github.com/apache/incubator-flagon-useralejs/tree/FLAGON-469 < > https://github.com/apache/incubator-flagon-useralejs/tree/FLAGON-469> > >> > >> As per docs, our workflow generally follows a pattern such that all > READMEs are updated prior to pushing new release candidates. Once released, > we update docs on the website at the same time that we update our website > to reflect new release links, etc. > >> > >> Currently, docs on UserALE.js — > http://flagon.incubator.apache.org/docs/useralejs/ < > http://flagon.incubator.apache.org/docs/useralejs/> — and the stack — > http://flagon.incubator.apache.org/docs/stack/ < > http://flagon.incubator.apache.org/docs/stack/> — are about up to date, > or lag by a patch release. The website is generally a good source for > guides and considerations, but our readmes are pretty solid. We’re always > willing to field asks for more specific documentation. > >> > >> We would love some help on our back-end. We’re lagging a bit as we’ve > not yet moved our examples up from ELK 6.8—>7.0. That’s on the near term > slate, as well as more documentation on clustering and indexing options. > But, we’d love PRs. > >> > >>> > >>> Thanks, > >>> Austin > >> > >> Thank You, Austin! I hope this is helpful! > >> > >
