These are great considerations and discussion. The super key question for me is the audience.
On Mon, Jun 8, 2015 at 11:08 AM, Chris Hillery <[email protected]> wrote: > If not, it may be worth taking a step back and asking what exactly the > problem is. I understand the general rule that "we don't want Asterix to be > broken", but what precisely does that mean? Is it acceptable that the tip > of the Asterix source branch is only guaranteed to build against the tip of > the Hyracks branch, for example? If not, why not? What audience are we > required to keep things working for at the source level, and what > expectations do they have? > > Ceej > aka Chris Hillery > > On Mon, Jun 8, 2015 at 2:06 AM, Chris Hillery <[email protected]> > wrote: > > > So, if we pushed these not-releases to the Nexus repo running at UCI, and > > devs pulled from there in preference to "official" repos, that would > solve > > the problem? > > > > Ceej > > aka Chris Hillery > > > > On Sun, Jun 7, 2015 at 7:29 PM, Ted Dunning <[email protected]> > wrote: > > > >> > >> If it is pushed to any wider audience than roughly the dev@ list, it is > >> a release. That definitely includes maven central. Artifacts in maven > are > >> convenience binaries and this not a release but they should be > traceable to > >> an exact source release. > >> > >> Sent from my iPhone > >> > >> > On Jun 7, 2015, at 19:10, Till Westmann <[email protected]> wrote: > >> > > >> > Hmm, good point. It doesn’t have to. One question might be if we can > >> push it to some maven repository, if it’s not an official release. > >> > But I think that should also be fine as long as we don’t push it to a > >> repository that claims to contain official releases. > >> > > >> > Some mentor input might be helpful on this as well :) > >> > > >> > Cheers, > >> > Till > >> > > >> >> On Jun 7, 2015, at 6:53 PM, Ildar Absalyamov < > >> [email protected]> wrote: > >> >> > >> >> Does version bump always mean full-fledged Apache release? We need > the > >> former just to resolve compile time dependencies. > >> >> > >> >>> On Jun 7, 2015, at 18:49, Till Westmann <[email protected]> wrote: > >> >>> > >> >>> In principle I agree with this, but creating a new release will be a > >> little more involved that just running maven, when we do this at the > ASF. > >> >>> To publish a new release we will have to vet and vote on the > release. > >> This takes at least 72 hours in the best case if we’re a TLP, the first > >> release candidate is great, and have enough people to vote. While we’re > >> still in the incubator, releasing will take a little longer as we also > have > >> to get enough votes for the release in the incubator. > >> >>> As I proposed earlier, it would be really good to go through the > full > >> release process once, before we decide how to structure our processes > and > >> infrastructure. > >> >>> > >> >>> Cheers, > >> >>> Till > >> >>> > >> >>>> On Jun 4, 2015, at 6:37 PM, Ildar Absalyamov < > >> [email protected]> wrote: > >> >>>> > >> >>>> I am with Chris on repository separation and I think that the > >> solution to the issue of Hyracks commits breaking Asterix build is using > >> release Hyracks versions instead of snapshot ones. Yes, that will > create a > >> frequent Hyracks releases (we will have to release it each time there > is a > >> change which spans both Hyracks & Asterix) and we have abandoned this > >> practice a while ago, but it seems that’s the only way to separate > projects > >> logically. > >> >>>> > >> >>>> Here are few examples to clear the picture. In all examples Hyracks > >> version is 4.5.6-Snapshot, Asterix version is 1.2.3-Snapshot (but it > >> depends on previous release version Hyracks 4.5.5): > >> >>>> 1) The changes span both Asterix & Hyracks. > >> >>>> First make sure that Asterix could depend on Hyracks 4.5.6-Snapshot > >> without API conflicts & switch Asterix dependency to 4.5.6-Snapshot. > >> >>>> Submit Gerrit review, once it is done as a part of git-asf script > >> commit changes, bump Hyracks version to 4.5.6, make Asterix depend on > 4.5.6 > >> and bump Hyracks to 4.5.7-Snapshot right after. > >> >>>> 2) The changes are located only in Hyracks. Regular review and > >> commit (with snapshot version) without any version bump. > >> >>>> 3) The changes are located only in Asterix. Regular review and > >> commit (with snapshot version) without any version bump. > >> >>>> > >> >>>> In this scenario Hyracks commit can never make Asterix build fail > >> (since it depends on a stable release) and it’s the responsibility of > the > >> first person, whose commits spans both repos to make sure that the > changes > >> in snapshot Hyracks version are properly merged. > >> >>>> > >> >>>> Regarding the Yingyi’s issue with Gerrit topics: could we modify > >> git-gerrit script so it would submit both Asterix & Hyracks reviews > >> (granted that the latter is needed), and link them together, setting the > >> proper topic? Gerrit seems to have API for changing that, right? > >> >>>> > >> >>>>> On Jun 4, 2015, at 15:45, Mike Carey <[email protected]> wrote: > >> >>>>> > >> >>>>> Just a quick high-level note from our nearest equivalent of the > >> pointy-haired Dilbert guy (aka me): What would be nice is to have > Hyracks > >> changes kick off tests of all "supported client projects" - AsterixDB, > >> VXQuery, maybe also Pregelix, IMRU, and possibly others in the future. > I > >> don't think we'll ever prevent such downstream things from being broken > >> unless we run their tests - so I would suggest that we need a mechanism > to > >> keep Hyracks changes from being permitted to happen without verifying > the > >> ongoing integrity of all "blessed" (priority 1) affected projects.... > We > >> could have an agreed upon list of such projects and tests for each.... > It > >> would be nice to have a "quick check" (hello world still works, basics > are > >> working) that was synchronously blocking of such changes, and at least a > >> daily verification that all's totally well (AFAWK) for them all. > >> >>>>> > >> >>>>> Not sure how this affects the still two-sided discussion... :-) > >> >>>>> > >> >>>>> Cheers, > >> >>>>> Mike > >> >>>>> > >> >>>>> > >> >>>>>> On 6/2/15 10:00 AM, Chris Hillery wrote: > >> >>>>>>> On Mon, Jun 1, 2015 at 9:46 PM, Yingyi Bu <[email protected]> > >> wrote: > >> >>>>>>> > >> >>>>>>> In my opinion, merging the repository doesn't break the > >> separation of > >> >>>>>>> hyracks and asterixdb, because the dependencies are controlled > by > >> mvn pom > >> >>>>>>> files. > >> >>>>>>> > >> >>>>>> That wasn't the separation I was talking about. I meant API > >> separation. As > >> >>>>>> it is now, when we make a change to both Asterix and Hyracks, we > >> are forced > >> >>>>>> to consider the API implications, or at least they are put out > >> there in a > >> >>>>>> very clear way that we need to look at. If we merge them, people > >> will > >> >>>>>> (rightly) treat the whole thing as one product, and there will be > >> no brakes > >> >>>>>> on making wide-ranging API changes. > >> >>>>>> > >> >>>>>> (As an aside: I don't trust Maven's pom files to do a good job of > >> keeping > >> >>>>>> the dependency management clean. In fact I trust it to do > >> precisely the > >> >>>>>> opposite, by making it both easier to screw up the dependencies > >> and harder > >> >>>>>> to update them in future.) > >> >>>>>> > >> >>>>>> Again, my point is this: If we truly believe that Hyracks is a > >> re-usable > >> >>>>>> component, it should be treated as such from source to build to > >> delivery. > >> >>>>>> By merging in Asterix, we are saying that Asterix is "more equal" > >> than > >> >>>>>> others Hyracks clients, to the point that we're tacitly willing > to > >> break > >> >>>>>> those other clients in favor of simplifying Asterix development. > >> If that is > >> >>>>>> a fair and true statement, well, then, sure, let's merge them. > >> >>>>>> > >> >>>>>> 1) It forces those hyracks-only changes to pass asterixdb > >> regression > >> >>>>>>> tests. Currently hyracks-only change are not verified by > >> asterixdb tests. > >> >>>>>>> > >> >>>>>> This is a good point, I will admit. However, I think this same > >> goal can be > >> >>>>>> met in other ways. My strong preference would be to create a set > >> of true > >> >>>>>> API tests inside of Hyracks, which both document and test the > >> external > >> >>>>>> Hyracks API. That will make API-breaking changes in future much > >> easier to > >> >>>>>> spot, and also make it clear when Asterix is using internal APIs > >> that it > >> >>>>>> should not. > >> >>>>>> > >> >>>>>> > >> >>>>>>> 2) On my local machine, I don't need to always install hyracks > >> and then > >> >>>>>>> verify asterixdb from time to time. Especially, switching > >> branches seems > >> >>>>>>> painful because the installed hyracks snapshot is overwritten > >> from time to > >> >>>>>>> time. > >> >>>>>>> > >> >>>>>> I haven't tried working on multiple Hyracks branches at the same > >> time, so I > >> >>>>>> haven't experienced this. This seems like a working method error, > >> though. > >> >>>>>> If you're working with two things that are "the same version" > >> (even if > >> >>>>>> that's a snapshot version), you'll need to use separate Maven > >> repositories > >> >>>>>> to install them. In fact, merging the two git repositories would > >> do nothing > >> >>>>>> to fix this problem, will it? If the proposal is to put the two > >> source > >> >>>>>> repositories in the same git repo but otherwise leave them > >> untouched, then > >> >>>>>> nothing would change in the build process. It's possible I'm > >> missing > >> >>>>>> something there, though. > >> >>>>>> > >> >>>>>> > >> >>>>>>> 3) I only need to make one code review request and one jenkins > >> job. > >> >>>>>>> Currently I need to manually change the topic of my asterixdb > >> gerrit CL > >> >>>>>>> every time before I update my hyracks CL, and then manually > >> schedule > >> >>>>>>> jenkins to run a new asterixdb job. If I forget to schedule the > >> jenkins > >> >>>>>>> job, the asterixdb CL is still shown to be "verified by > jenkins". > >> >>>>>>> > >> >>>>>> This is a problem, but it's a problem in commit validation, not > in > >> the > >> >>>>>> source. Modifying the source to work around these issues is still > >> a bad > >> >>>>>> idea IMHO. > >> >>>>>> > >> >>>>>> The "change-topic" issue could be fixed with a bit of development > >> work > >> >>>>>> (have the topic point to a change, rather than a specific > patchset > >> on the > >> >>>>>> change, so you only need to set it once, for instance). > >> >>>>>> > >> >>>>>> As for manually scheduling Asterix Jenkins jobs, that sounds like > >> it's only > >> >>>>>> a problem where your Hyracks change breaks an existing public > API. > >> That > >> >>>>>> would be obviated by having true API testing inside of Hyracks, > >> which is > >> >>>>>> something that we should have regardless of any decisions about > >> source > >> >>>>>> locations. > >> >>>>>> > >> >>>>>> In summary / repeating myself again: yes, we have some problems > >> because > >> >>>>>> Hyracks and Asterix are in seperate repositories. But those > >> problems are > >> >>>>>> pointing out true issues with our development and processes. > >> Merging the > >> >>>>>> repositories isn't fixing those problems, it's sweeping them > under > >> the rug. > >> >>>>>> Long term we would be much better off to identify, isolate, and > >> fix the > >> >>>>>> problems themselves. > >> >>>>>> > >> >>>>>> Ceej > >> >>>>>> aka Chris Hillery > >> >>>>>> > >> >>>>> > >> >>>> > >> >>>> Best regards, > >> >>>> Ildar > >> >>>> > >> >>> > >> >> > >> >> Best regards, > >> >> Ildar > >> >> > >> > > >> > > > > >
