Re: Migration of git repository

[Till Westmann]

I'm also not too fond about merging - especially with my VXQuery hat on.

Another $.02,
Till

On 1 Jun 2015, at 17:15, Chris Hillery wrote:

My $.02 - no, we shouldn't.

Two main reasons:

1. If we're serious about Hyracks being a re-usable component of other
products, it makes sense to dogfood that in Asterixdb. If there are
problems keeping Hyracks separate from Asterix or keeping Hyracks with
clean interfaces, this forces us to address them.

2. We only just recently took the initiative to take Pregelix and

Hiversterix *out* of the same repository, and that was because they were specifically causing us problems as components of the same build. (There were issues of competing dependency versions with Ian's YARN work, as well

as several spurious pregelix test failures, as I recall.) At a bare

minimum, we cannot merge those projects back in without re-researching and

addressing those problems.

What benefits would we gain by merging them? I honestly don't agree with

Yingyi's suggestion that it would make building, bug-fixing, and code

review much simpler. At best it would help a bit on those occasions when a change spans Hyracks and Asterix, and again, IMHO that is something that *should* require additional thought and oversight. As for build and test,

my feeling is that it will make it considerably harder, or at the very
least slower, simply due to doubling the Maven overhead.

I do not feel that merging the projects to either fit in better with
Apache, or to game the Apache popularity indexes, is a good trade-off.

Ceej
aka Chris Hillery

On Mon, Jun 1, 2015 at 12:02 PM, Yingyi Bu <buyin...@gmail.com> wrote:

Hi folks,

Should we merge hyracks, asterixdb, and potentially pregelix/imru into the same repository? It will make build, fix, and code review process

much simpler.

An example is that everything built on top of Spark lives in the same repository: https://github.com/apache/spark. That's also why Spark is

the most active Apache project now, due to its commit frequency.
 Does anyone have concerns for merging the hyracks and asterixdb
repositories?
 Thanks!

Best,
Yingyi

On Wed, Apr 22, 2015 at 10:13 PM, Till Westmann <ti...@apache.org> wrote:

Ok, let’s find out what is the “more work” part before we decide :)

We should already have the SGA (as it’s part of the SGA that Mike sent in) and it seemed to me that all we’re need to do “later” (e.g. next

week/month) would be to

a) vote on bringing it into AsterixDB (that would be an incubator vote I

assume) and
b) asking infra for another git repository.
So the extra work would be the vote on the incubator list.
Is that right or is there something else we’d need to do?

Cheers,
Till

On Apr 22, 2015, at 10:04 PM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

Hey Mike and team,

Thanks for bringing this to the list. I think these are precisely
the type of conversations that we want to have here at the ASF and
as part of our Incubating project. Having these discussions in the

community here at the ASF (which is now the Apache AsterixDB community)

is great.

My opinion - it’s fine either way. I’m happy if you guys want to
bring Pregelix into the code base here via AsterixDB. It’s easily
reversible and incremental. If you want to spin out Pregelix later
as its own TLP and it’s shown to have its own community we can
file a board resolution to do that. Heck, nothing stops us from
graduating 2 Incubator projects=>TLPs out of this effort even in
the Incubator. That’s fine. If you want to wait and bring it in
later, it will definitely be more work - so let’s call a spade a
spade there. But if you want to do that that’s fine too.

My personal recommendation - bring it in - won’t hurt and we can
always pivot in the ways above later.

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Michael Carey <mjca...@ics.uci.edu>
Date: Tuesday, April 21, 2015 at 11:49 AM
To: Chris Mattmann <chris.a.mattm...@jpl.nasa.gov>, Till Westmann
<t...@westmann.org>
Cc: Chris Hillery <chill...@lambda.nu>, Ian Maxon <ima...@uci.edu>,
Yingyi
Bu <buyin...@gmail.com>, "dev@asterixdb.incubator.apache.org"
<dev@asterixdb.incubator.apache.org>
Subject: Re: Migration of git repository

Sure! Let me clarify the issue for everyone (and broaden the question).

One of the technical by-products of the AsterixDB project is a graph

analytics package called Pregelix - as the name suggests, it is a "knock

off" of Pregel, as are packages like Giraph.  What's unique about
Pregelix is that it actually scales without OOM'ing

- under the covers it uses database join processing techniques. You can

find out more about it by visiting
http://pregelix.ics.uci.edu/ and/or by skimming the attached paper -
check out the experimental results compared to other popular

alternatives. Anyway, we have made it freely available (as we do all of

our AsterixDB-related

research products) and we were thinking that we should simply include it under the AsterixDB project - kind of like Spark has subprojects for SQL,

streams, graphs, etc.  As a result, I listed it on the list of
transferred artifacts when I sent in the licensing
form the other day.  (So we at least have that step done.)  Its code

conntributors have been a small subset of the AsterixDB team; it was a

small sub-project, basically.  (Mostly just Yingyi Bu!)

Pregelix is kind of a sibling of Apache VXQuery in that its runtime is

based on Hyracks but it hasn't otherwise been AsterixDB-dependent.

However, we have just finished teaching it to read/write directly from

AsterixDB native storage - instead of just HDFS
- so now it has an AsterixDB dependency, and we are using it as a

driving example of how to couple AsterixDB to other analytic engines.

Rather than going through another exercise to open-source this
separately, it seemed like we could take this approach.

Thoughts?
Cheers,
Mike


On 4/21/15 7:45 AM, Mattmann, Chris A (3980) wrote:


Yes, in fact, this whole conversations should be happening on
the dev list. OK for me to CC them on my reply?

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: "Michael J. Carey" <mjca...@ics.uci.edu>
<mailto:mjca...@ics.uci.edu <mjca...@ics.uci.edu>>
Date: Tuesday, April 21, 2015 at 3:13 AM
To: Till Westmann <t...@westmann.org> <mailto:t...@westmann.org
<t...@westmann.org>>
Cc: Chris Hillery <chill...@lambda.nu> <mailto:chill...@lambda.nu
<chill...@lambda.nu>>, Ian

Maxon <ima...@uci.edu> <mailto:ima...@uci.edu <ima...@uci.edu>>, Yingyi Bu <buyin...@gmail.com> <mailto:buyin...@gmail.com <buyin...@gmail.com>>,

Chris Mattmann

<chris.a.mattm...@jpl.nasa.gov> <mailto:chris.a.mattm...@jpl.nasa.gov

<chris.a.mattm...@jpl.nasa.gov>>
Subject: Re: Migration of git repository

+ Yingyi on the Pregelix Q. Should we also ask Chris M for advice on

that?
On Apr 20, 2015 4:23 PM, "Till Westmann" <t...@westmann.org>
<mailto:t...@westmann.org <t...@westmann.org>> wrote:

Hi Ian,


That’s a good question - and I don’t know the answer.
We’ve got 2 repos so far:
https://issues.apache.org/jira/browse/INFRA-9212https://issues.apache.org/
jira/browse/INFRA-9306
so we should have space for Hyracks and AsterixDB.

I think that there’s an open questions about Pregelix, but maybe that

shouldn’t keep us from going ahead.

I further think that it would be great if you could send an e-mail to

dev@asterixdb.incubator.apache.org<
mailto:dev@asterixdb.incubator.apache.o
<dev@asterixdb.incubator.apache.o>
rg> <mailto:dev@asterixdb.incubator.apache.org
<dev@asterixdb.incubator.apache.org>> and ask if it’s ok to
import

our git repo(s) or if something else needs to be done first. (I could

send that e-mail as well, but it would be great if there were more
non-Till e0mails on the list :) )


Cheers,
Till


On Apr 20, 2015, at 4:07 PM, Ian Maxon <ima...@uci.edu>
<mailto:ima...@uci.edu <ima...@uci.edu>> wrote:

Hi Mike, Chris and Till,

Since (I think?) the paperwork for the software grant is done now, should I copy our GC branches over to the ASF git repositories now ( as well as

making it a mirror in the Gerrit commit hook script)?


Thanks,
- Ian