I think your later point about using a project release version, from the example of using other projects, is a valid one. To exactly that point, a community member (Otto) brought up an issue/bug they found through testing that they were previously unaware of and did not find documented. Which was argued would be confusing to someone wanting to use a published release. We discussed the implications of this bug/feature gap. And incidentally, it sounds like some people see full dev as more useful than just a dev box, others do not, independent of what we chose to name it. That came from our discussion about it.
The expectation I had from my discussion with the contributors was that this fix for aggregation was ready. So to your point about whether it belonged or not, I'm inclined to say yes, had it been ready. I personally don't like this feature gap in full dev. It seems Otto agrees, and Casey at the very least sees it as enough of an issue to gate us from 0.8. New information about that feature has changed my mind about what to do about it in the short term. I think we should move forward. Our vote landed 2-2. We are having a discussion about what to do with the release. This is that discussion. On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjl...@gmail.com> wrote: > @Mike > I have a different question: Why is this enough to consider delaying a > release in the first place for a fairly involved fix? > > There was a discuss thread, where the general agreement was that we had > enough value to do a release (Over a month ago. And more things have gone > into master since then). There's a good number of fixes, and not just > trivial ones either. The general consensus here seems to be that the > management UI issue is fairly minor for a point release (after all, there's > been multiple people who think option 2 is sufficient), but becomes > important if we want to release a minor version. The question I asked > myself about this was ""Does this issue detract enough value that a release > isn't worthwhile?" and my answer was, and still is, "No, we have enough > value to do a meaningful release". > > I'm fine with delaying or cancelling a release because we find issues that > are severe enough or we don't think there's enough value anymore, but to be > entirely honest, I'm absolutely shocked this issue has blown up so much. > However, if you want to have a discuss thread to reevaluate if it's > worthwhile to do a release, go for it. The communities' calculus on the > "Does this issue detract enough value that a release isn't worthwhile?" may > be different than mine. > > Having said all that, to a large extent, I think you're right. It really > doesn't matter* that much* if we release next week or the week after or > whenever. But at the same time I personally get super frustrated when I go > to use a project, find a bug, it's already known and fixed, but it just > never puts out a released version. Every cutoff is largely arbitrary, but > I think getting our improvements and fixes out there is important. One of > the things we've done fairly well is put out releases at a fairly decent > cadence for a project this large. I really don't want to set the precedent > of just increasingly pushing out point releases for stuff like this. > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <n...@nickallen.org> wrote: > > > I think any open source project needs to strive to cut releases > regularly. > > This is healthy for the project and community. It gets new features and > > functionality out to the community so we can get feedback, find what is > > working and what is not, iterate and improve. You probably agree with > > this. > > > > While releasing this week or next may not matter in the grand scheme, if > we > > want to cut releases regularly, then we need to bear down and just do it. > > Case in point, I opened the initial discussion for this release on March > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks later. > > > > -- > > [1] > > > > > https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E > > > > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > As a more general question, can I ask why we're feeling pressure to > push > > > out a release in the first place? Again, I'm happy to continue with > > option > > > 2. Let's move forward and get out the release. But is there a reason > why > > we > > > think it has to get out now, versus next week, or the week after? Otto > > > pointed out a legitimate issue, dev environment or not, and I'm unclear > > why > > > we have an issue with waiting for the fix. There's no pressure on this, > > > imho. > > > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > > > I remember this now, but I’m not sure how I would have related this > to > > a > > > > parser aggregation pr honestly. > > > > > > > > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) > > > wrote: > > > > > > > > Here's a link to the ngrx discussion thread from a few months back: > > > > > > > > > > > > > > https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E > > > > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler <ottobackwa...@gmail.com> > > > > wrote: > > > > > > > > > If you can find a link in the archives for that thread, it would > > really > > > > > help. > > > > > > > > > > I don’t think sending them up as one sensor would work…. as > something > > > > > quick. I think it is an interesting idea from a higher level that > > would > > > > > need some more thought though ( IE: what if every sensor in the ui > > was > > > a > > > > > sensor group, and the existing entries where just groups of 1 ). > > > > > > > > > > As far as I can see, we have brought up the idea of a release > > > ourselves, > > > > I > > > > > don’t see why we don’t just swarm this issue and get it right then > > > > release. > > > > > > > > > > > > > > > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) > > wrote: > > > > > > > > > > In PR#1360 we introduced a new state management strategy involving > a > > > new > > > > > module called Ngrx. We had a discussion thread on this a few months > > ago > > > > and > > > > > we successfully convinced you about the benefits. This is one of > the > > > > > reasons why this PR is going to be still huge after cleaning up the > > > > commit > > > > > history. After you having a look at the changes and the feature > > itself, > > > > > there's likely have questions about why certain parts work as they > > do. > > > > The > > > > > thing what I'd like to point out is that, yes, it probably takes > more > > > > time > > > > > to get it in. > > > > > > > > > > In order to being able to release the RC, wouldn't it be an easy > and > > > > quick > > > > > fix on the backend if it sent the aggregated parsers to the client > as > > > > they > > > > > were one sensor? It's just an idea, it might be wrong, but at least > > we > > > > > shouldn't have to wait until the aforementioned PR gets ready to be > > > > merged > > > > > to the master. > > > > > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet <justinjl...@gmail.com> > > > > wrote: > > > > > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker > for > > > > 0.8.0. > > > > > > #3 seems like a total waste of time and effort. > > > > > > > > > > > > The wall of text version: > > > > > > I agree this isn't "just the wrong thing shown", but for > completely > > > > > > different reasons. > > > > > > > > > > > > To be extremely clear about what the problem is: Our "dev" > > > environment > > > > > > (whose very name implies the audience is develops) uses a > > > > > performance-based > > > > > > advanced feature to ensure that all our of sample flows are > > regularly > > > > run > > > > > > and produce data. This feature has a bare minimal implementation > to > > > be > > > > > > enabled via Ambari, which it currently is by default. This is > > because > > > > of > > > > > > the limited resources available that previously resulted in us > > > turning > > > > > off > > > > > > Yaf, and therefore testing it during regular full dev runs. Right > > now > > > > > > however, this feature is not exposed through the management UI, > and > > > > > > therefore it isn't obvious what the implications are. Am I > missing > > > > > anything > > > > > > here? > > > > > > > > > > > > For users actually choosing to use the parser aggregation feature > > in > > > a > > > > > > non-full-dev environment, I'd expect substantially more care to > be > > > > > involved > > > > > > given the lack of easy configuration for it (after all, why would > > you > > > > > > bother running the aggregated parser alongside the regular > parser? > > > This > > > > > > could be more explicitly stated, but again that feels like a doc > > > > problem. > > > > > > Right now I could essentially provide two of the same parser and > > > create > > > > > the > > > > > > same problem, so right now aggregation is only special because it > > > runs > > > > on > > > > > > dev by default). This is, in my opinion, primarily a first > > impression > > > > > > problem and likely one of many areas that could use improved > > > > > documentation. > > > > > > > > > > > > Quite frankly, I think the issue pointed out here could mostly be > > > > > resolved > > > > > > by documenting how the current aggregation is done in dev, and > > > telling > > > > > how > > > > > > to change it. Especially for a 0.x.1 release, which is primarily > > bug > > > > > > fixes. As can be inferred from my vote, I don't think this > problem > > > is a > > > > > > problem that needs solving in a point release. I would support > > > > improving > > > > > > the documentation, both full-dev and for aggregation in general > for > > > the > > > > > > 0.7.1 point release, while making a 0.8.0 release contingent upon > > the > > > > > > outstanding PRs to enable it in the management UI. > > > > > > > > > > > > There are a couple deeper issues, imo, that I care substantially > > more > > > > > about > > > > > > than this in particular > > > > > > * The dev environment is being used as our intro for users, > because > > > > it's > > > > > > convenient for us to not maintain more environments (which has > > been a > > > > > major > > > > > > pain point in the past). Worse, the dev environment strongly > > implies > > > > it's > > > > > > for Metron developers, rather than people looking to build on top > > of > > > > > > Metron. We need an actual strategy for providing end users a > clean > > > > > > impression of Metron (this could be clarifying what the > > expectations > > > of > > > > > > full dev are, renaming it to something like "full-demo", > something > > > more > > > > > > involved, etc.). This is something that we've needed for awhile > in > > > > > general, > > > > > > and includes larger topics like improving our website, > potentially > > > > > > improving the site book, actually publishing our Javadocs > somewhere > > > so > > > > > > people can develop things easier, publishing out info about > Stellar > > > > > > functions in a better manner, etc. > > > > > > * The fact that parsers are handled in Ambari at all. It's awful > > and > > > > > leads > > > > > > to situations like this. To the best of my knowledge, once we can > > do > > > > > > chaining and aggregation in the Management UI, we should be able > to > > > > > > entirely divorce these two overlapping domains. I'd love to see > > > parsers > > > > > > ripped out of Ambari, then full-dev manages all the setup via > REST. > > > At > > > > > that > > > > > > point, we can easily tell everyone to just use the management UI. > > > > > > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler < > > ottobackwa...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I think it would help if the full consequences of having the UI > > > show > > > > > the > > > > > > > wrong status where listed. > > > > > > > > > > > > > > Someone trying metron, will, by default , see the wrong thing > in > > > the > > > > UI > > > > > > for > > > > > > > the ONLY sensors they have that are running and doing data. > > > > > > > > > > > > > > What happens when they try to start them to make them work? > One, > > > two > > > > or > > > > > > > all? > > > > > > > What happens when he edits them or try to add transformations? > > One, > > > > two > > > > > > or > > > > > > > all? > > > > > > > What other things can you do with the sensors in the ui? What > > > > happens? > > > > > > > > > > > > > > Are we recommending aggregation on the list and elsewhere for > > > users? > > > > > Are > > > > > > > we recommending something that is going to ensure they get into > > > this > > > > > > > situation? > > > > > > > > > > > > > > I think this is more than ‘just the wrong thing shown’ in the > ui. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic ( > > > > > > > michael.miklav...@gmail.com) wrote: > > > > > > > > > > > > > > The vote for RC1 did not pass and I'd like to kickstart some > > > > discussion > > > > > > > about what we should do. > > > > > > > > > > > > > > I started taking a look at PR#1360 and it looks like this isn't > > > quite > > > > > as > > > > > > > close to being able go in as I had originally expected. I want > to > > > > talk > > > > > > > about options here. It seems to me that we can: > > > > > > > > > > > > > > 1. Wait for PR#1360 to go in, but this is likely going to take > > more > > > > > time > > > > > > > than originally anticipated > > > > > > > 2. Accept the issue in full dev, but add some notes in the > > > developer > > > > > > > docs about the current feature gap and why sensors aren't > showing > > > > > status > > > > > > in > > > > > > > the management UI when aggregation is enabled. > > > > > > > 3. Find some other workable UI solution. > > > > > > > 4. Other option? > > > > > > > > > > > > > > All things considered, I'm personally leaning towards #2 in the > > > > > > short-term, > > > > > > > but I think we should probably talk about this a bit before > > > deciding > > > > > what > > > > > > > RC2 should be. > > > > > > > > > > > > > > Best, > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > >
