PR is up. I added the doc change to the metron-deployment README since this serves as the gateway doc for all the VM instances. All of which would be affected by the feature gap.
https://github.com/apache/metron/pull/1398 On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Here's the ticket I created to track it, which also references the Jira > for the new UI feature. > https://issues.apache.org/jira/browse/METRON-2100 > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> :-) >> >> I expect to have #2 out sometime today. >> >> On Thu, May 2, 2019, 12:11 PM Justin Leet <justinjl...@gmail.com> wrote: >> >>> > >>> > I personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. >>> > >>> >>> +1 on all of this. I don't like it either. >>> >>> >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> >>> >>> I'm going to be honest, my response was a combination of misreading what >>> you said and thinking you were proposing delaying the release more >>> seriously and feeling a bit blindsided by a perceived move from the >>> initial >>> "take more time than originally anticipated" (which in my head I took as >>> a >>> couple days) to "versus next week, or the week after" (where delaying >>> things weeks is something I personally would like not buried so far down >>> in >>> the thread). Totally my bad, sorry about that. >>> >>> Other than that, it sounds like we're pretty much in agreement. >>> >>> Here's my current understanding of the state and consensus as of right >>> now >>> (which is subject to change as more discussion happens): >>> >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and #3 >>> for 0.8.0. >>> - I don't believe I've seen an explicit response from Otto on what >>> he >>> thinks about doing this, and from a personal perspective like to >>> see what >>> his opinion is as the person who originally brought it up. >>> - Mike said he's going to kick out a PR that addresses #2 >>> - After that undergoes the normal review process and is merged, we >>> proceed normally and cut RC2. >>> >>> >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < >>> michael.miklav...@gmail.com> wrote: >>> >>> > I think your later point about using a project release version, from >>> the >>> > example of using other projects, is a valid one. To exactly that >>> point, a >>> > community member (Otto) brought up an issue/bug they found through >>> testing >>> > that they were previously unaware of and did not find documented. >>> Which was >>> > argued would be confusing to someone wanting to use a published >>> release. We >>> > discussed the implications of this bug/feature gap. And incidentally, >>> it >>> > sounds like some people see full dev as more useful than just a dev >>> box, >>> > others do not, independent of what we chose to name it. That came from >>> our >>> > discussion about it. >>> > >>> > The expectation I had from my discussion with the contributors was that >>> > this fix for aggregation was ready. So to your point about whether it >>> > belonged or not, I'm inclined to say yes, had it been ready. I >>> personally >>> > don't like this feature gap in full dev. It seems Otto agrees, and >>> Casey at >>> > the very least sees it as enough of an issue to gate us from 0.8. New >>> > information about that feature has changed my mind about what to do >>> about >>> > it in the short term. I think we should move forward. >>> > >>> > Our vote landed 2-2. We are having a discussion about what to do with >>> the >>> > release. This is that discussion. >>> > >>> > On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjl...@gmail.com> >>> wrote: >>> > >>> > > @Mike >>> > > I have a different question: Why is this enough to consider delaying >>> a >>> > > release in the first place for a fairly involved fix? >>> > > >>> > > There was a discuss thread, where the general agreement was that we >>> had >>> > > enough value to do a release (Over a month ago. And more things have >>> gone >>> > > into master since then). There's a good number of fixes, and not just >>> > > trivial ones either. The general consensus here seems to be that the >>> > > management UI issue is fairly minor for a point release (after all, >>> > there's >>> > > been multiple people who think option 2 is sufficient), but becomes >>> > > important if we want to release a minor version. The question I asked >>> > > myself about this was ""Does this issue detract enough value that a >>> > release >>> > > isn't worthwhile?" and my answer was, and still is, "No, we have >>> enough >>> > > value to do a meaningful release". >>> > > >>> > > I'm fine with delaying or cancelling a release because we find issues >>> > that >>> > > are severe enough or we don't think there's enough value anymore, >>> but to >>> > be >>> > > entirely honest, I'm absolutely shocked this issue has blown up so >>> much. >>> > > However, if you want to have a discuss thread to reevaluate if it's >>> > > worthwhile to do a release, go for it. The communities' calculus on >>> the >>> > > "Does this issue detract enough value that a release isn't >>> worthwhile?" >>> > may >>> > > be different than mine. >>> > > >>> > > Having said all that, to a large extent, I think you're right. It >>> really >>> > > doesn't matter* that much* if we release next week or the week after >>> or >>> > > whenever. But at the same time I personally get super frustrated >>> when I >>> > go >>> > > to use a project, find a bug, it's already known and fixed, but it >>> just >>> > > never puts out a released version. Every cutoff is largely >>> arbitrary, >>> > but >>> > > I think getting our improvements and fixes out there is important. >>> One of >>> > > the things we've done fairly well is put out releases at a fairly >>> decent >>> > > cadence for a project this large. I really don't want to set the >>> > precedent >>> > > of just increasingly pushing out point releases for stuff like this. >>> > > >>> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <n...@nickallen.org> >>> wrote: >>> > > >>> > > > I think any open source project needs to strive to cut releases >>> > > regularly. >>> > > > This is healthy for the project and community. It gets new >>> features >>> > and >>> > > > functionality out to the community so we can get feedback, find >>> what is >>> > > > working and what is not, iterate and improve. You probably agree >>> with >>> > > > this. >>> > > > >>> > > > While releasing this week or next may not matter in the grand >>> scheme, >>> > if >>> > > we >>> > > > want to cut releases regularly, then we need to bear down and just >>> do >>> > it. >>> > > > Case in point, I opened the initial discussion for this release on >>> > March >>> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks >>> > later. >>> > > > >>> > > > -- >>> > > > [1] >>> > > > >>> > > > >>> > > >>> > >>> https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E >>> > > > >>> > > > >>> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < >>> > > > michael.miklav...@gmail.com> wrote: >>> > > > >>> > > > > As a more general question, can I ask why we're feeling pressure >>> to >>> > > push >>> > > > > out a release in the first place? Again, I'm happy to continue >>> with >>> > > > option >>> > > > > 2. Let's move forward and get out the release. But is there a >>> reason >>> > > why >>> > > > we >>> > > > > think it has to get out now, versus next week, or the week after? >>> > Otto >>> > > > > pointed out a legitimate issue, dev environment or not, and I'm >>> > unclear >>> > > > why >>> > > > > we have an issue with waiting for the fix. There's no pressure on >>> > this, >>> > > > > imho. >>> > > > > >>> > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler < >>> ottobackwa...@gmail.com> >>> > > > wrote: >>> > > > > >>> > > > > > I remember this now, but I’m not sure how I would have related >>> this >>> > > to >>> > > > a >>> > > > > > parser aggregation pr honestly. >>> > > > > > >>> > > > > > >>> > > > > > On May 2, 2019 at 07:54:13, Shane Ardell ( >>> shane.m.ard...@gmail.com >>> > ) >>> > > > > wrote: >>> > > > > > >>> > > > > > Here's a link to the ngrx discussion thread from a few months >>> back: >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E >>> > > > > > >>> > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler < >>> > ottobackwa...@gmail.com> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > If you can find a link in the archives for that thread, it >>> would >>> > > > really >>> > > > > > > help. >>> > > > > > > >>> > > > > > > I don’t think sending them up as one sensor would work…. as >>> > > something >>> > > > > > > quick. I think it is an interesting idea from a higher level >>> that >>> > > > would >>> > > > > > > need some more thought though ( IE: what if every sensor in >>> the >>> > ui >>> > > > was >>> > > > > a >>> > > > > > > sensor group, and the existing entries where just groups of >>> 1 ). >>> > > > > > > >>> > > > > > > As far as I can see, we have brought up the idea of a release >>> > > > > ourselves, >>> > > > > > I >>> > > > > > > don’t see why we don’t just swarm this issue and get it right >>> > then >>> > > > > > release. >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor ( >>> ftamas.m...@gmail.com) >>> > > > wrote: >>> > > > > > > >>> > > > > > > In PR#1360 we introduced a new state management strategy >>> > involving >>> > > a >>> > > > > new >>> > > > > > > module called Ngrx. We had a discussion thread on this a few >>> > months >>> > > > ago >>> > > > > > and >>> > > > > > > we successfully convinced you about the benefits. This is >>> one of >>> > > the >>> > > > > > > reasons why this PR is going to be still huge after cleaning >>> up >>> > the >>> > > > > > commit >>> > > > > > > history. After you having a look at the changes and the >>> feature >>> > > > itself, >>> > > > > > > there's likely have questions about why certain parts work as >>> > they >>> > > > do. >>> > > > > > The >>> > > > > > > thing what I'd like to point out is that, yes, it probably >>> takes >>> > > more >>> > > > > > time >>> > > > > > > to get it in. >>> > > > > > > >>> > > > > > > In order to being able to release the RC, wouldn't it be an >>> easy >>> > > and >>> > > > > > quick >>> > > > > > > fix on the backend if it sent the aggregated parsers to the >>> > client >>> > > as >>> > > > > > they >>> > > > > > > were one sensor? It's just an idea, it might be wrong, but at >>> > least >>> > > > we >>> > > > > > > shouldn't have to wait until the aforementioned PR gets >>> ready to >>> > be >>> > > > > > merged >>> > > > > > > to the master. >>> > > > > > > >>> > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet < >>> > justinjl...@gmail.com> >>> > > > > > wrote: >>> > > > > > > >>> > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a >>> blocker >>> > > for >>> > > > > > 0.8.0. >>> > > > > > > > #3 seems like a total waste of time and effort. >>> > > > > > > > >>> > > > > > > > The wall of text version: >>> > > > > > > > I agree this isn't "just the wrong thing shown", but for >>> > > completely >>> > > > > > > > different reasons. >>> > > > > > > > >>> > > > > > > > To be extremely clear about what the problem is: Our "dev" >>> > > > > environment >>> > > > > > > > (whose very name implies the audience is develops) uses a >>> > > > > > > performance-based >>> > > > > > > > advanced feature to ensure that all our of sample flows are >>> > > > regularly >>> > > > > > run >>> > > > > > > > and produce data. This feature has a bare minimal >>> > implementation >>> > > to >>> > > > > be >>> > > > > > > > enabled via Ambari, which it currently is by default. This >>> is >>> > > > because >>> > > > > > of >>> > > > > > > > the limited resources available that previously resulted >>> in us >>> > > > > turning >>> > > > > > > off >>> > > > > > > > Yaf, and therefore testing it during regular full dev runs. >>> > Right >>> > > > now >>> > > > > > > > however, this feature is not exposed through the >>> management UI, >>> > > and >>> > > > > > > > therefore it isn't obvious what the implications are. Am I >>> > > missing >>> > > > > > > anything >>> > > > > > > > here? >>> > > > > > > > >>> > > > > > > > For users actually choosing to use the parser aggregation >>> > feature >>> > > > in >>> > > > > a >>> > > > > > > > non-full-dev environment, I'd expect substantially more >>> care to >>> > > be >>> > > > > > > involved >>> > > > > > > > given the lack of easy configuration for it (after all, why >>> > would >>> > > > you >>> > > > > > > > bother running the aggregated parser alongside the regular >>> > > parser? >>> > > > > This >>> > > > > > > > could be more explicitly stated, but again that feels like >>> a >>> > doc >>> > > > > > problem. >>> > > > > > > > Right now I could essentially provide two of the same >>> parser >>> > and >>> > > > > create >>> > > > > > > the >>> > > > > > > > same problem, so right now aggregation is only special >>> because >>> > it >>> > > > > runs >>> > > > > > on >>> > > > > > > > dev by default). This is, in my opinion, primarily a first >>> > > > impression >>> > > > > > > > problem and likely one of many areas that could use >>> improved >>> > > > > > > documentation. >>> > > > > > > > >>> > > > > > > > Quite frankly, I think the issue pointed out here could >>> mostly >>> > be >>> > > > > > > resolved >>> > > > > > > > by documenting how the current aggregation is done in dev, >>> and >>> > > > > telling >>> > > > > > > how >>> > > > > > > > to change it. Especially for a 0.x.1 release, which is >>> > primarily >>> > > > bug >>> > > > > > > > fixes. As can be inferred from my vote, I don't think this >>> > > problem >>> > > > > is a >>> > > > > > > > problem that needs solving in a point release. I would >>> support >>> > > > > > improving >>> > > > > > > > the documentation, both full-dev and for aggregation in >>> general >>> > > for >>> > > > > the >>> > > > > > > > 0.7.1 point release, while making a 0.8.0 release >>> contingent >>> > upon >>> > > > the >>> > > > > > > > outstanding PRs to enable it in the management UI. >>> > > > > > > > >>> > > > > > > > There are a couple deeper issues, imo, that I care >>> > substantially >>> > > > more >>> > > > > > > about >>> > > > > > > > than this in particular >>> > > > > > > > * The dev environment is being used as our intro for users, >>> > > because >>> > > > > > it's >>> > > > > > > > convenient for us to not maintain more environments (which >>> has >>> > > > been a >>> > > > > > > major >>> > > > > > > > pain point in the past). Worse, the dev environment >>> strongly >>> > > > implies >>> > > > > > it's >>> > > > > > > > for Metron developers, rather than people looking to build >>> on >>> > top >>> > > > of >>> > > > > > > > Metron. We need an actual strategy for providing end users >>> a >>> > > clean >>> > > > > > > > impression of Metron (this could be clarifying what the >>> > > > expectations >>> > > > > of >>> > > > > > > > full dev are, renaming it to something like "full-demo", >>> > > something >>> > > > > more >>> > > > > > > > involved, etc.). This is something that we've needed for >>> awhile >>> > > in >>> > > > > > > general, >>> > > > > > > > and includes larger topics like improving our website, >>> > > potentially >>> > > > > > > > improving the site book, actually publishing our Javadocs >>> > > somewhere >>> > > > > so >>> > > > > > > > people can develop things easier, publishing out info about >>> > > Stellar >>> > > > > > > > functions in a better manner, etc. >>> > > > > > > > * The fact that parsers are handled in Ambari at all. It's >>> > awful >>> > > > and >>> > > > > > > leads >>> > > > > > > > to situations like this. To the best of my knowledge, once >>> we >>> > can >>> > > > do >>> > > > > > > > chaining and aggregation in the Management UI, we should be >>> > able >>> > > to >>> > > > > > > > entirely divorce these two overlapping domains. I'd love >>> to see >>> > > > > parsers >>> > > > > > > > ripped out of Ambari, then full-dev manages all the setup >>> via >>> > > REST. >>> > > > > At >>> > > > > > > that >>> > > > > > > > point, we can easily tell everyone to just use the >>> management >>> > UI. >>> > > > > > > > >>> > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler < >>> > > > ottobackwa...@gmail.com> >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > I think it would help if the full consequences of having >>> the >>> > UI >>> > > > > show >>> > > > > > > the >>> > > > > > > > > wrong status where listed. >>> > > > > > > > > >>> > > > > > > > > Someone trying metron, will, by default , see the wrong >>> thing >>> > > in >>> > > > > the >>> > > > > > UI >>> > > > > > > > for >>> > > > > > > > > the ONLY sensors they have that are running and doing >>> data. >>> > > > > > > > > >>> > > > > > > > > What happens when they try to start them to make them >>> work? >>> > > One, >>> > > > > two >>> > > > > > or >>> > > > > > > > > all? >>> > > > > > > > > What happens when he edits them or try to add >>> > transformations? >>> > > > One, >>> > > > > > two >>> > > > > > > > or >>> > > > > > > > > all? >>> > > > > > > > > What other things can you do with the sensors in the ui? >>> What >>> > > > > > happens? >>> > > > > > > > > >>> > > > > > > > > Are we recommending aggregation on the list and >>> elsewhere for >>> > > > > users? >>> > > > > > > Are >>> > > > > > > > > we recommending something that is going to ensure they >>> get >>> > into >>> > > > > this >>> > > > > > > > > situation? >>> > > > > > > > > >>> > > > > > > > > I think this is more than ‘just the wrong thing shown’ >>> in the >>> > > ui. >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic ( >>> > > > > > > > > michael.miklav...@gmail.com) wrote: >>> > > > > > > > > >>> > > > > > > > > The vote for RC1 did not pass and I'd like to kickstart >>> some >>> > > > > > discussion >>> > > > > > > > > about what we should do. >>> > > > > > > > > >>> > > > > > > > > I started taking a look at PR#1360 and it looks like this >>> > isn't >>> > > > > quite >>> > > > > > > as >>> > > > > > > > > close to being able go in as I had originally expected. I >>> > want >>> > > to >>> > > > > > talk >>> > > > > > > > > about options here. It seems to me that we can: >>> > > > > > > > > >>> > > > > > > > > 1. Wait for PR#1360 to go in, but this is likely going to >>> > take >>> > > > more >>> > > > > > > time >>> > > > > > > > > than originally anticipated >>> > > > > > > > > 2. Accept the issue in full dev, but add some notes in >>> the >>> > > > > developer >>> > > > > > > > > docs about the current feature gap and why sensors aren't >>> > > showing >>> > > > > > > status >>> > > > > > > > in >>> > > > > > > > > the management UI when aggregation is enabled. >>> > > > > > > > > 3. Find some other workable UI solution. >>> > > > > > > > > 4. Other option? >>> > > > > > > > > >>> > > > > > > > > All things considered, I'm personally leaning towards #2 >>> in >>> > the >>> > > > > > > > short-term, >>> > > > > > > > > but I think we should probably talk about this a bit >>> before >>> > > > > deciding >>> > > > > > > what >>> > > > > > > > > RC2 should be. >>> > > > > > > > > >>> > > > > > > > > Best, >>> > > > > > > > > Mike >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >>
